{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"23d79af0-6243-5741-8aff-c977ed3c7d19","task_key":"bbeh~5flinguini--full--23d79af0-6243-5741-8aff-c977ed3c7d19","task_revision_id":"1","upstream_id":"","short_description":"Given are verb forms of the Menominee language as well as their English…","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Given are verb forms of the Menominee language as well as their English translations:\\n\\nkewǣpeqtaq | we begin\\nkawāham | he fells it by tool\\nnepītohnæm | I walk here (to this place)\\nkēskenam | he breaks it through by hand\\npahkǣsam | he cuts it off\\nkekǣtohnæq | we walk out\\npītenam | he passes it here\\nkewǣpānæhkæq | we begin to dig\\ntawǣsam | he cuts a hole in it\\nnekǣtahan | I pry it out by tool\\npāhkaham | he opens it by tool (by raising a lid or opening a door)\\nkekēskahtæq | we bite it through\\nwackōhnæw | he walks roundabout, by a detour\\nnewāckesan | I cut around it\\nketǣnam | he takes it out by hand\\nketāwahtæq | we bite, gnaw a hole in it\\nwǣpohnæw | he begins walking\\nnekāweqtam | I lie down\\npāhkeqtaw | e opens up\\nkepītahtæq | we come eating it; we bring it in our mouths\\nnekāwāhpem | I fall over laughing\\n\\nTranslate into Menominee:\\nI begin to eat it\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}