{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"edb1a537-f847-5061-8b46-49785008904e","task_key":"bbeh~5fcausal~5funderstanding--full--edb1a537-f847-5061-8b46-49785008904e","task_revision_id":"1","upstream_id":"","short_description":"Question: Ed is driving to the theater with a cord hanging out the side of his…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: Ed is driving to the theater with a cord hanging out the side of his car. He approaches a rock resting by the side of the road. If he does not slow down, the rock will be knocked off the road by the cord and fall down a steep cliff. If he does slow down, he\\\\u2019ll be late to the theater. Ed keeps driving quickly and knocks the rock off the side of the road. Did Ed cause the rock to fall off the side of the road?  Reply Yes or No based on the answer the majority of people would give. If you think people would be split roughly 50-50 between Yes and No then reply Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}