{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f887d666-98ec-5111-b01d-0f96bfc6dae9","task_key":"formal~5ffallacies--main--f887d666-98ec-5111-b01d-0f96bfc6dae9","task_revision_id":"1","upstream_id":"","short_description":"\"Here comes a perfectly valid argument: Not being an ingredient of Eye Shadow…","config":"formal_fallacies","split":"main","body":"{\"input\":\"\\\"Here comes a perfectly valid argument: Not being an ingredient of Eye Shadow (Gold) is sufficient for not being an ingredient of real fit lipstick 9. Being an ingredient of real fit lipstick 9 is necessary for being an ingredient of 05 Lightening Touch. Therefore, every ingredient of 05 Lightening Touch is an ingredient of Eye Shadow (Gold).\\\"\\nIs the argument, given the explicitly stated premises, deductively valid or invalid?\\nOptions:\\n- valid \\n- invalid\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}