{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7682c9c7-7862-5ac0-b8d0-d8b95c7614ee","task_key":"bbeh~5fspatial~5freasoning--full--7682c9c7-7862-5ac0-b8d0-d8b95c7614ee","task_revision_id":"1","upstream_id":"","short_description":"You have been given a circular grid consisting of 12 connected dots. Starting…","config":"bbeh_spatial_reasoning","split":"full","body":"{\"input\":\"You have been given a circular grid consisting of 12 connected dots. Starting from a vertex, you will move along the edges of the circular grid. Initially, you are positioned on the dot that's located at the top of the grid, where you find an Irish Water Spaniel. You move around the ring by 6 steps in a clockwise direction, where you find a television. You move around the ring by 2 steps in a counter-clockwise direction, where you find a wallet. You move around the ring by 7 steps in a clockwise direction, where you find a castle. You move around the ring by 6 steps in a clockwise direction, where you find a bulbul. You move around the ring by 5 steps in a clockwise direction, where you find a wing. You move around the ring by 2 steps in a counter-clockwise direction, where you find a Petri dish. You move around the ring by 6 steps in a counter-clockwise direction, where you find a station wagon. You move around the ring by 6 steps in a counter-clockwise direction. What will you find? Your final answer must be only the object name (e.g., laptop). If you have not previously seen the object at the vertex, your final answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}