{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"47ff0032-33df-5557-900a-1236d50760d4","task_key":"bbeh~5fspatial~5freasoning--full--47ff0032-33df-5557-900a-1236d50760d4","task_revision_id":"1","upstream_id":"","short_description":"You have been given a tree structure with 86 nodes. trumpet and thorn are…","config":"bbeh_spatial_reasoning","split":"full","body":"{\"input\":\"You have been given a tree structure with 86 nodes. trumpet and thorn are siblings. mosque has no grandchildren. prompt has no children. play has two children; augmented and snail. tablet has 4 children: coat, audio, prompt, and steak. file has no grandchildren. regression has no children. friend has no children. evolution has no children. nail has 4 children: chicken, battery, puzzle, and brush. venture has no grandchildren. freedom has no grandchildren. stamp has no grandchildren. icecream has no children. soap has 4 children: dustpan, eggplant, evolution, and stamp. present has two children; icecream and empowerment. empowerment has no grandchildren. eggplant has no children. snail has no children. coat has no grandchildren. frequency has 4 children: commit, resistance, mosque, and bird. trumpet has 14 grandchildren: soap, venture, frequency, and milk (whose parent is cat), politics, tablet, and nail (whose parent is conclusion), television and challenge (whose parent is sculpture), and dance, present, play, corn, and actuator (whose parent is signal). politics has no grandchildren but has 3 children: sugar, butterfly, and sandwich. tensor has no children. actuator has two children; conditioner and tensor. stage has no grandchildren. shine has no grandchildren. brush has no grandchildren. dustpan has no grandchildren. audio has no grandchildren. output has 5 children: emotion, dust, coin, purpose, and profile. optimization has 8 grandchildren: crab, washer, file, and stage (whose parent is paper) and lemon, deploy, innovation, and potato (whose parent is map). goose has no children. potato has no grandchildren. challenge has 4 children: architecture, interface, freedom, and regression. beauty has two children; output and sofa. steak has no children. crab has no grandchildren. emotion has no grandchildren. lemon has no grandchildren. puzzle has no grandchildren. profile has no children. corn has no grandchildren but has 4 children: sunglasses, curtain, squash, and melon. chicken has no grandchildren. deploy has no grandchildren. purpose has no grandchildren. milk has no grandchildren but has 3 children: video, movement, and ticket. skateboard has no grandchildren. television has no grandchildren but has 4 children: webpage, pen, message, and lace. commit has no children. coin has no children. sofa has 4 children: friend, shine, skateboard, and glitter. interface has no grandchildren. architecture has no children. glitter has no grandchildren. bird has no grandchildren. conditioner has no grandchildren. thorn has two children; beauty and optimization. improvement has no grandchildren. innovation has no children. dust has no grandchildren. washer has no children. resistance has no children. dance has two children; goose and improvement. augmented has no children. battery has no children. What is the cousin of the sugar? Your final answer must be only the object name (e.g., laptop). If there are multiple objects, provide them as a comma-separated list in alphabetical order (e.g., laptop, mug). Your final answer must be only the object name (e.g., laptop). If you have not previously seen the object at the vertex, your final answer must be \\\"unknown\\\".\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}