{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d6bff480-a39b-586a-a6f8-754ea965c2dd","task_key":"formal~5ffallacies--main--d6bff480-a39b-586a-a6f8-754ea965c2dd","task_revision_id":"1","upstream_id":"","short_description":"\"Is Siri a stepsister of Mary? Is Susan related to Kate? In large families, it…","config":"formal_fallacies","split":"main","body":"{\"input\":\"\\\"Is Siri a stepsister of Mary? Is Susan related to Kate? In large families, it is sometimes difficult to keep track of all one's relatives. The following argument seeks to clarify some such relations: To start with, being a close friend of Catherine is necessary for not being a classmate of Leslie. Now, being a granddaughter of Courtney is necessary for not being a classmate of Leslie. Moreover, whoever is not a classmate of Mary is however a close friend of Catherine or a granddaughter of Courtney. In consequence, whoever is not a classmate of Leslie is a classmate of Mary.\\\"\\nIs the argument, given the explicitly stated premises, deductively valid or invalid?\\nOptions:\\n- valid \\n- invalid\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}