{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2e18748e-8996-5d9e-9066-98fbe1f50fa9","task_key":"bbeh~5fcausal~5funderstanding--full--2e18748e-8996-5d9e-9066-98fbe1f50fa9","task_revision_id":"1","upstream_id":"","short_description":"Question: Alice intends to put lethal poison into Carol's water. However, Alice…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: Alice intends to put lethal poison into Carol's water. However, Alice does not put lethal poison into Carol's water (¬AP). Bob puts an antidote into Carol's water (BA). The water is lethal (L), if the poison is added without the addition of an antidote. If Carol would consume the lethal water she would die (CD). Carol consumes her water (CC). Carol does not die (¬CD). Is Alice not putting lethal poison into the water a sufficient cause for Carol's survival? Reply based on the answer a logician would give.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}