{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"de593cea-e1f8-590b-9ef8-46f994be8c27","task_key":"bbeh~5fcausal~5funderstanding--full--de593cea-e1f8-590b-9ef8-46f994be8c27","task_revision_id":"1","upstream_id":"","short_description":"Question: A group of students who lived on the same floor of a dormitory…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: A group of students who lived on the same floor of a dormitory obtained a copy of the final exam for their biology class. The students cheated on the test. One student, John Granger, did not go along with the group. Granger deviates from the norm and refuses to cheat on the test. The biology class comprises 80 students and is graded on a curve such that 20 people will receive a grade of A, 20 a grade of B, 20 a grade of C, and 20 students will receive a D. Granger's score was the 20th-highest score in the class, which means he was the last student to receive a grade of A. The 21st student was a pre-med student who received a B and, as a result, missed the GPA cutoff she needed to get into the medical school she was hoping for by .07 GPA points. Did Granger cause the student to fail to meet the medical school cutoff?  Reply Yes or No based on the answer the majority of people would give. If you think people would be split roughly 50-50 between Yes and No then reply Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}