{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"33327c5c-e652-5565-bbd9-7a9c278a61a4","task_key":"bbeh~5flinguini--full--33327c5c-e652-5565-bbd9-7a9c278a61a4","task_revision_id":"1","upstream_id":"","short_description":"Here are some words in Tundra Yukaghir and their English translations in…","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Here are some words in Tundra Yukaghir and their English translations in arbitrary order:\\n\\n1. ilennime\\n2. joqonnime\\n3. saancohoje\\n4. johudawur\\n5. ilenlegul\\n6. cireme\\n7. johul\\n8. aariinmøŋer\\n9. joqodile\\n10. møŋer\\n11. ciremennime\\n12. joqoncohoje\\n13. saadoŋoj\\n14. uoduo\\n15. oŋoj\\n16. aariinjohul\\n17. uodawur\\n18. joqol\\n\\nA. gunshot\\nB. wooden box\\nC. nose\\nD. bird\\nE. Yakut knife\\nF. deer feed\\nG. bag\\nH. rifle’s muzzle\\nI. horse\\nJ. nose case\\nK. wooden house\\nL. grandchild\\nM. thunder\\nN. Yakut person\\nO. cradle\\nP. herd of deer\\nQ. wooden knife\\nR. nest\\n\\nThe word ewce means ‘tip, point’. Translate into English:\\nile\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}