{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7ed587d1-b0bd-5a19-8667-4a8ecfd0bab5","task_key":"formal~5ffallacies--main--7ed587d1-b0bd-5a19-8667-4a8ecfd0bab5","task_revision_id":"1","upstream_id":"","short_description":"\"Here comes a perfectly valid argument: First of all, being an ingredient of…","config":"formal_fallacies","split":"main","body":"{\"input\":\"\\\"Here comes a perfectly valid argument: First of all, being an ingredient of Gorgeous is necessary for not being an ingredient of Baked Eyeshadow. Next, something is not an ingredient of Gorgeous and, in addition, not an ingredient of Coverage Duo or not an ingredient of Deternined. Hence, there is something that is an ingredient of Baked Eyeshadow and not both an ingredient of Coverage Duo and an ingredient of Deternined.\\\"\\nIs the argument, given the explicitly stated premises, deductively valid or invalid?\\nOptions:\\n- valid \\n- invalid\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}