{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8ef71a53-a928-5a8e-a8e3-b1bc55d0ba7c","task_key":"bbeh~5flinguini--full--8ef71a53-a928-5a8e-a8e3-b1bc55d0ba7c","task_revision_id":"1","upstream_id":"","short_description":"Proto-Chamic is the theorized common ancestor of all the languages in the Chamic…","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Proto-Chamic is the theorized common ancestor of all the languages in the Chamic group within the Austronesian family. It was reconstructed by linguists by comparing its many present-day descendants, among which are Phan Rang Cham and Tsat. Here are some words in Proto-Chamic, Phan Rang Cham and Tsat, and their meanings:\\n\\nProto-Chamic | Phan Rang Cham | Tsat | meaning\\n*ʔikaːn | ikan | kaːn³³ | fish\\n*dikiʔ | takiʔ | kiʔ⁴² | few\\n*dilah | talàh | la⁵⁵ | tongue\\n*jalaːn | calàn | laːn¹¹ | road\\n*lima | limɨ | ma³³ | 5\\n*lanah | linɨh | na⁵⁵ | pus\\n*dua | lapan | tàlipan | paːnʔ⁴² | 8\\n*labaːt | lipàʔ | phaːʔ⁴² | to walk\\n*bara | pirà | phia¹¹ | shoulder\\n*bahrow | piròw | phiə¹¹ | new\\n*bulow | pilòw | phiə¹¹ | body hair\\n*paley | paley | piai³³ | village\\n*masam | mɨtham | saːnʔ⁴² | vinegar\\n*basah | pathah | sa⁵⁵ | wet\\n*bəsey | pathey | sai¹¹ | iron\\n*mata | mɨta | ta³³ | eye\\n*rataːk | rataʔ | taːʔ²⁴ | beans\\n*hatay | hatay | taːiʔ⁴² | liver\\n*paday | patày | thaːiʔ⁴² | paddy rice\\n*padam | patàm | thaːnʔ⁴² | to extinguish\\n*ʔiduŋ | itùng | thuŋ¹¹ | nose\\n*batuk | patuʔ | tuʔ⁴² | to cough\\n*maray | mɨray | zaːiʔ⁴² | to come\\n*ʔurat | uraʔ | zaʔ²⁴ | tendon\\n*rək | rəʔ | zəʔ²⁴ | grass\\n*bala | pilà | - | widowed\\n\\nFill in the masked element indicated with (7), assuming the (1), (2), ... elements are masked out elements.\\n\\nProto-Chamic | Phan Rang Cham | Tsat | meaning\\n*kulit | kaliʔ | (1) | skin\\n*hitam | hatam | (2) | black\\n*bubah | papàh | (3) | mouth\\n*ʔikat | (4) | (5) to tie\\n*dəpa | (6) | (7) fathom\\n*matay | (8) | (9) to die\\n*dalam | (10) | (11) inside\\n*labuh | (12) | (13) to fall down\\n*bulaːn | (14) | (15) moon\\n- | pala | (16) | to plant\\n- | tapùh | (17) | to ransom\\n- | tatà | (18) | chest\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}