{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b1d3c440-5bf5-54c1-9eb7-c3c3f3babafc","task_key":"bbeh~5flinguini--full--b1d3c440-5bf5-54c1-9eb7-c3c3f3babafc","task_revision_id":"1","upstream_id":"","short_description":"Here are some words and verb roots in West Tarangan. Each is also given in its…","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Here are some words and verb roots in West Tarangan. Each is also given in its so-called reduplicated form in two dialects of the language. The reduplicated form is used as a participle or adjective.\\n\\n  |  | North dialect | Coast dialect\\ndakɛru | they scrape | darkɛru | dakɛrukɛru\\nrɔbik | it is blunt | rɔbrɔbik | rɔbirɔbik\\nalema | on the right | amlema | alemalema\\nmakay | you climb | makmakay | mamakay\\napúk | other | akpuk | apukpuk\\nlɔpay | cold | lɔplɔpay | lɔpalɔpay\\npayláwana | he is talkative | paylawlawana | paylalawana\\nkalpaŋir | it is obstinate | kalpaŋpaŋir | kalpapaŋir\\ngarkɔ́w | to be orphaned | garkɔwkɔw | garkɔwkɔw\\nɛylɛka | he plays | ɛylɛklɛka | ɛylɛlɛka\\ndubɛmna | he is seventh | dumbɛmna | dubɛmbɛmna\\nmaylɛwa | (a species of) tree | maylɛwlɛwa | maylɛlɛwa\\nmatay | his eye | matmatay | matamatay\\nmɔna | in front | mɔnmɔna | mɔnamɔna\\nbɛbar | to be scared | bɛbɛbar | bɛbɛbar\\njaŋil | it is rotten | jaŋjaŋil | jajaŋil\\nletna | he is male | letletna | letletna\\nɛrnɔnaw | he crawls | ɛrnɔnɔnaw | ɛrnɔnɔnaw\\n\\nFill in the masked element indicated with (17), assuming the (1), (2), ... elements are masked out elements.\\n\\n  |  | North dialect | Coast dialect\\nɛtaleŋa | he hears | (1) | (2)\\njaga | to guard | (3) | (4)\\ngasírana | she is old | (5) | (6)\\ndaramota | they vomit | (7) | (8)\\npɔwna | he is smelly | (9) | (10)\\nɛrtɔpa | he washes | (11) | (12)\\ndabuka | they dream | (13) | (14)\\nŋuŋim | it is damp | (15) | (16)\\nigɔ́ŋ | he sells | (17) | (18)\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}