{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"949ef68b-e60d-5223-953a-846bb7911a34","task_key":"formal~5ffallacies--main--949ef68b-e60d-5223-953a-846bb7911a34","task_revision_id":"1","upstream_id":"","short_description":"\"Is Siri a stepsister of Mary? Is Susan related to Kate? In large families, it…","config":"formal_fallacies","split":"main","body":"{\"input\":\"\\\"Is Siri a stepsister of Mary? Is Susan related to Kate? In large families, it is sometimes difficult to keep track of all one's relatives. The following argument seeks to clarify some such relations: To begin with, every daughter of Milagros is not a great-grandmother of Courtney or not an ancestor of Christy. Moreover, nobody is neither a great-grandmother of Courtney nor a niece of Lynn. Finally, being a niece of Lynn is necessary for not being an ancestor of Christy. We may conclude that every daughter of Milagros is a niece of Lynn.\\\"\\nIs the argument, given the explicitly stated premises, deductively valid or invalid?\\nOptions:\\n- valid \\n- invalid\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}