{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9157fd9d-3b27-53f2-b42c-ded6b74c7202","task_key":"bbeh~5fcausal~5funderstanding--full--9157fd9d-3b27-53f2-b42c-ded6b74c7202","task_revision_id":"1","upstream_id":"","short_description":"Question: There are a left and a right window. Alice and Bob both order Carol to…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: There are a left and a right window. Alice and Bob both order Carol to fire at the left window. Carol fires at the left window, shattering it. Commands from Alice always trump commands from Bob (e.g. if Bob would have ordered to fire at the right window, Carol would still have fired at the left one). Without a command Carol would not have fired at all. Is Alice ordering Carol to fire a necessary cause for the window shattering? Reply based on the answer a logician would give.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}