{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a64191d7-adcd-51f7-90b8-0275be20419c","task_key":"bbeh~5fcausal~5funderstanding--full--a64191d7-adcd-51f7-90b8-0275be20419c","task_revision_id":"1","upstream_id":"","short_description":"Question: Jack is driving five sick people to the hospital with a cord hanging…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: Jack is driving five sick people to the hospital with a cord hanging out the side of his car. He approaches a rock climber resting by the side of the road. If he does not slow down, the climber will be knocked off the road by the cord and fall down a steep cliff. If he does slow down, the five sick people will die before they reach the hospital. Jack keeps driving quickly and knocks the rock climber off the side of the road. Did Jack cause the climber to fall off the side of the road?  Reply Yes or No based on the answer the majority of people would give. If you think people would be split roughly 50-50 between Yes and No then reply Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}