{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"40adba4e-b681-524b-a83d-b62e05c5a265","task_key":"formal~5ffallacies--main--40adba4e-b681-524b-a83d-b62e05c5a265","task_revision_id":"1","upstream_id":"","short_description":"\"Is Fred a cousin of Robert? Is Joe related to Bob? In large families, it is…","config":"formal_fallacies","split":"main","body":"{\"input\":\"\\\"Is Fred a cousin of Robert? Is Joe related to Bob? In large families, it is sometimes difficult to keep track of all one's relatives. The following argument seeks to clarify some such relations: First of all, whoever is not a half-brother of Steven is a great-grandfather of Craig. Next, there is somebody who is a great-grandfather of Craig and, in addition, not an ancestor of Jordon or not a stepbrother of Gustavo. It follows that not every half-brother of Steven is both an ancestor of Jordon and a stepbrother of Gustavo.\\\"\\nIs the argument, given the explicitly stated premises, deductively valid or invalid?\\nOptions:\\n- valid \\n- invalid\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}