{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"19372d5b-4e92-59e4-ac30-8ce53a712b39","task_key":"bbeh~5fcausal~5funderstanding--full--19372d5b-4e92-59e4-ac30-8ce53a712b39","task_revision_id":"1","upstream_id":"","short_description":"Question: Tom works for a demolition company, and today he is demolishing a…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: Tom works for a demolition company, and today he is demolishing a building by implosion. The building will implode automatically at 5:00 PM if both the safety switch is off and knob A is switched on. At 4:00 PM, the safety switch is off, and knob A is off. At that time, Tom checks to see if knob A is on, and he sees that it's off. So, he changes the position of knob A to the on position. Because the building would implode if both the safety switch is off and knob A is switched on, the building implodes at 5:00 PM. Did the building implode because Tom changed the position of knob A?  Reply Yes or No based on the answer the majority of people would give. If you think people would be split roughly 50-50 between Yes and No then reply Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}