{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5010b450-4245-5de2-bc61-8070c6c21d99","task_key":"bbeh~5fcausal~5funderstanding--full--5010b450-4245-5de2-bc61-8070c6c21d99","task_revision_id":"1","upstream_id":"","short_description":"Question: Carol is alive (CA). Alice puts a harmless antidote in Carol's water…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: Carol is alive (CA). Alice puts a harmless antidote in Carol's water (AA). Adding antidote to the water protects it against poison (WS - 'water save'). If Alice puts the antidote into Carol’s water, Bob will poison the water (BP). Adding poison to an unprotected water makes it toxic (WT). Carol consumes her water and survives (CS). If Carol would drink toxic water she would die. Is Alice putting the harmless antidote in the water a necessary cause for Carol's survival? Reply based on the answer a logician would give.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}