{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8b4906c8-4098-5d64-8038-805ad7fbf141","task_key":"bbeh~5fcausal~5funderstanding--full--8b4906c8-4098-5d64-8038-805ad7fbf141","task_revision_id":"1","upstream_id":"","short_description":"Question: In Bash Ball, opposing teams try to score points in the other team's…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: In Bash Ball, opposing teams try to score points in the other team's zone by entering it. On each team, there are two types of players: Bashers and Runners. During the game, Bashers remain in the same spot, trying to hit the other team's Runners with balls, and Runners try to enter the other team's zone without getting hit. Zach, Harmon and a few other friends are on a team playing Bash Ball against another team. Zach is a Basher and Harmon is a Runner. At one point in the game, Zach spots one of the other team's Bashers behind a tree. Later on, Harmon passes by Zach. Knowing that he is allowed to talk to Runners, Zach does not tell Harmon where the other team's Basher is. Soon after, the Basher that Zach spotted hits Harmon with a Bash Ball. Harmon is out for the remainder of the game. Was Harmon out of the game because Zach did not tell Harmon where the other team's Basher was?  Reply Yes or No based on the answer the majority of people would give. If you think people would be split roughly 50-50 between Yes and No then reply Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}