{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5ed38ebd-3df5-52d3-b01e-fcf8aad387c6","task_key":"formal~5ffallacies--main--5ed38ebd-3df5-52d3-b01e-fcf8aad387c6","task_revision_id":"1","upstream_id":"","short_description":"\"Is Titanium oxide an ingredient of my washing power? Which chemicals does my…","config":"formal_fallacies","split":"main","body":"{\"input\":\"\\\"Is Titanium oxide an ingredient of my washing power? Which chemicals does my perfume contain? It is really difficult to keep track of all chemicals one is regularly exposed to. The following argument seeks to clarify some such relations: First, it is not the case that Coffee bean extract is an ingredient of Go 365 Shampoo. Second, everything that is an ingredient of Chickadee is an ingredient of Go 365 Shampoo, too. So, necessarily, it is not the case that Coffee bean extract is an ingredient of Chickadee.\\\"\\nIs the argument, given the explicitly stated premises, deductively valid or invalid?\\nOptions:\\n- valid \\n- invalid\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}