{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"946c1b91-f6c9-50c4-9773-16352f4fc682","task_key":"formal~5ffallacies--main--946c1b91-f6c9-50c4-9773-16352f4fc682","task_revision_id":"1","upstream_id":"","short_description":"\"Consumer research aims at understanding whether users of some products also…","config":"formal_fallacies","split":"main","body":"{\"input\":\"\\\"Consumer research aims at understanding whether users of some products also tend to consume other ones, or not. The following argument seeks to clarify some such relations: First of all, Donald is an infrequent user of CHI shampoo. Next, every infrequent user of CHI shampoo and every loyal buyer of Lifebuoy soap is an infrequent user of Natural Bar soap. It follows that Donald is an infrequent user of Natural Bar soap.\\\"\\nIs the argument, given the explicitly stated premises, deductively valid or invalid?\\nOptions:\\n- valid \\n- invalid\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}