{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8bcbcd8c-176b-5a2f-97e0-487c58b4cf13","task_key":"bbeh~5fspatial~5freasoning--full--8bcbcd8c-176b-5a2f-97e0-487c58b4cf13","task_revision_id":"1","upstream_id":"","short_description":"You have been given an equilateral triangular tile map consisting of N rows,…","config":"bbeh_spatial_reasoning","split":"full","body":"{\"input\":\"You have been given an equilateral triangular tile map consisting of N rows, where the first row has one tile, the second row has three tiles,  ..., and the N-th row has 2*N-1 triangle-shaped tiles. There is a unique object placed at each vertex. Starting from a vertex, you will move along the edges of these tiles and see the object at each vertex. You are initially at the top corner where you see a fig. Then you move down-left for one step and see a cake. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move down-left for one step and see a soap. Then you move down-left for one step and see a statue. Then you move up-right for one step. Then you move down-left for one step. Then you move right for two steps. Then you move up-left for one step. Then you move left for one step. Then you move down-right for one step and see a television. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move down-right for one step and see a grape. Then you move down-left for one step. Then you move down-left for one step. Then you move up-right for one step. Then you move up-right for one step. Then you move left for one step and see a raspberry. Then you move down-left for one step. Then you move down-right for one step. Then you move up-right for one step. Then you move up-left for one step. Then you move left for one step and see a rug. Then you move down-left for two steps. Then you move up-right for one step. Then you move up-right for one step. Then you move down-right for one step and see a towel. Then you move left for one step and see a bed. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move down-right for one step and see a paper. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move down-right for one step and see a potato. Then you move down-left for one step. Then you move down-right for one step. Then you move up-right for one step. Then you move up-left for one step. Then you move down-right for one step and see a cucumber. Then you move down-right for one step and see a desk. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move down-right for one step and see a fox. Then you move down-left for two steps. Then you move up-right for one step. Then you move up-right for one step. Then you move left for one step and see a stone. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move up-left for one step and see a football. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move right for one step. Then you move left for one step. Then you move down-left for one step. Then you move down-right for one step. Then you move up-right for one step. Then you move up-left for one step. Then you move up-left for one step and see a peanut. Then you move down-left for two steps. Then you move up-right for one step. Then you move up-right for one step. Then you move down-right for one step. Then you move down-left for one step. Then you move up-right for one step. Then you move down-right for one step. Then you don't move for a while. Then you move up-right for one step. Then you don't move for a while. Then you move up-right for one step and see a flower. Then you don't move for a while. Then you move up-left for one step and see a toy. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for two steps. Then you move up-left for one step and see a bread. Then you don't move for a while. Then you move down-left for one step. Then you move down-left for one step. Then you don't move for a while. Then you move up-right for one step. Then you move down-left for one step. Then you move right for one step. Then you move up-left for one step. Then you move down-left for one step. Then you move down-left for one step. Then you move up-right for one step. Then you move down-left for one step and see a printer. Then you move down-left for one step. Then you move down-left for one step. Then you move up-right for one step. Then you move up-right for one step. Then you move down-left for one step and see a bike. Then you move down-left for one step. Then you move up-right for one step. Then you move left for one step and see a banana. Then you move down-left for one step. Then you move down-right for one step. Then you move up-right for one step. Then you move up-left for one step. Then you move up-left for one step and see a lychee. Then you move down-left for two steps. Then you move up-right for one step. Then you move up-right for one step. Then you move down-right for one step. Then you move down-left for two steps. Then you move up-right for one step. Then you move up-right for one step. Then you move left for one step and see a spoon. Then you move down-left for two steps. Then you move up-right for one step. Then you move up-right for one step. Then you move right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for two steps. Then you move up-right for one step and see a tiger. Then you move down-left for one step. Then you move right for two steps. Then you move up-left for one step. Then you move left for one step. Then you move up-right for one step and see a pie. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for two steps. Then you move left for one step and see a bean. Then you don't move for a while. Then you move up-right for one step and see a tree. Then you move down-left for one step. Then you move down-left for one step. Then you move up-right for one step. Then you move up-right for one step. Then you move right for one step and see a shampoo. Then you move down-left for one step. Then you move right for two steps. Then you move up-left for one step. Then you move left for one step. Then you move up-right for one step. Then you move down-left for one step. Then you move right for one step. Then you move up-left for one step. Then you move up-right for one step. Then you move down-left for one step. Then you move down-right for one step. Then you move up-right for one step. Then you move up-left for one step. Then you move down-left for one step. Then you don't move for a while. Then you move down-right for one step. Then you don't move for a while. Then you move down-right for one step. Then you don't move for a while. Then you move left for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move down-right for one step. Then you move up-left for three steps. Then you move down-right for one step. Then you move down-left for two steps. Then you move up-right for one step. Then you move up-right for one step. Then you move down-right for one step. Then you move down-left for one step. Then you move down-left for one step. Then you move up-right for one step. Then you move up-right for one step. What will you find?  Your final answer must be only the object name (e.g., laptop). If you have not previously seen the object at the vertex, your final answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}