{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cd498d0c-6ae7-54f6-9ad9-143fca515b34","task_key":"bbeh~5fcausal~5funderstanding--full--cd498d0c-6ae7-54f6-9ad9-143fca515b34","task_revision_id":"1","upstream_id":"","short_description":"Question: A bear and a hedgehog were shown a box full of colored pencils. Both…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: A bear and a hedgehog were shown a box full of colored pencils. Both hedgehogs and bears were allowed to take pencils. Both animals alternately took pencils out of the box six times. At last, the hedgehog and the bear came to the box and simultaneously took a pencil. A short time later, another animal, a polar bear, approached the box seeking a pencil to do its homework. However, there were no pencils left in the box. Polar bear cannot finish his homework. Did the hedgehog cause the problem?  Reply Yes or No based on the answer the majority of people would give. If you think people would be split roughly 50-50 between Yes and No then reply Ambiguous.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}