{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fb10235f-aad3-5a62-90a7-f421c68ad9dc","task_key":"bbeh~5fcausal~5funderstanding--full--fb10235f-aad3-5a62-90a7-f421c68ad9dc","task_revision_id":"1","upstream_id":"","short_description":"Question: Janet is an employee in the factory. She works in the maintenance…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: Janet is an employee in the factory. She works in the maintenance department where she monitors the stability of all machines. Since she works in the maintenance department, she knows how to grease and oil all of the machines in the factory. It is her responsibility to put oil into the machines. Kate is also an employee in the factory. She works in the human resources department where she monitors the salaries of all employees. While she works in the human resources department, she knows how to grease and oil all of the machines in the factory. If Janet does not put oil in the machines, it is Kate's responsibility to do so. On June 1st, Janet forgot to put oil into the machine. Kate also did not put oil in the machine. The machine broke down a few days later. Did the machine break down because Kate did not put oil in the machine?  Reply Yes or No based on the answer the majority of people would give. If you think people would be split roughly 50-50 between Yes and No then reply Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}