{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"29879025-c7f1-567c-9f41-88648e57df41","task_key":"bbeh~5flinguini--full--29879025-c7f1-567c-9f41-88648e57df41","task_revision_id":"1","upstream_id":"","short_description":"Here are some words in Yidiny and their English translations:","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Here are some words in Yidiny and their English translations:\\n\\ngudaːga | dog\\nbuɲa  | woman\\nwaguːɟa  | man\\nmuyubara  | stranger\\ngaɟagimbaːgu  | for a white man\\nbamagimbal  | without a person\\nbamaːgu | for a person\\nbimbiːbi | another father\\nmularigu | for an initiated man\\nmularini | of an initiated man\\nbuɲaːm | from a woman\\ngudagabi | another dog\\ngaɟagimbaːm | from a white man\\nbiɲɟiːngu | for a hornet\\nbimbiːn | of a father\\nmuɟam | mother\\nbiɲɟiːnmu | from a hornet\\nmaɟuːrbi | another frog\\nbuɲagimbal | without a woman\\nbaɟigalni | of a tortoise\\nɟuduːlumuɟay | with a pigeon\\nmugaɽumu | from a fishing net\\nwaŋaːlgu | for a boomerang\\n\\nThe mark “ː” indicates that the preceding vowel is long.\\n\\nTranslate into Yidiny:\\nof a pigeon\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}