{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"33ed8b81-89f8-54bb-a6d5-83067e7f3a15","task_key":"bbeh~5fcausal~5funderstanding--full--33ed8b81-89f8-54bb-a6d5-83067e7f3a15","task_revision_id":"1","upstream_id":"","short_description":"Question: There is a test tube on the left and a test tube on the right. Sam and…","config":"bbeh_causal_understanding","split":"full","body":"{\"input\":\"Question: There is a test tube on the left and a test tube on the right. Sam and Riya both order Frank to break the left test tube. Frank throws the left test tube, breaking it. Commands from Sam always trump commands from Riya (e.g. if Riya would have ordered to break the right test tube, Frank would still have thrown the left one). Without a command Frank would not have acted at all. Is Sam's order a sufficeint cause for the test tube to break? Reply based on the answer a logician would give.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}