{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"df1bedbb-7293-5408-b538-97b4c7fc6bc9","task_key":"bbeh~5flinguini--full--df1bedbb-7293-5408-b538-97b4c7fc6bc9","task_revision_id":"1","upstream_id":"","short_description":"Here are some words in Zuni and their English translations in arbitrary order:","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Here are some words in Zuni and their English translations in arbitrary order:\\n\\n1. mo:chikwa datda:we\\n2. haya:we\\n3. doko:ko mo’le\\n4. ma:chikwa\\n5. mu:chikwa\\n6. mulonne ≈ mu’le\\n7. mowe\\n8. ha:k’yawe\\n9. doko:ko shiwe\\n10. mo:chikwa mulo:chikwa\\nA. chicken meat\\nB. tea (drink)\\nC. sugar\\nD. loaf of bread\\nE. beads\\nF. peach pie\\nG. piles of leaves\\nH. chicken egg\\nI. peach trees\\nJ. chocolate bar\\n\\nHere are some more words in Zuni and their English translations, again in arbitrary order:\\n\\n11. ade muwe\\n12. beyanne\\n13. homa:we\\n14. ma:k’yawe\\n15. ma:k’ose\\n16. ma’le\\n17. homayanne\\n18. mi’le\\n19. ade k’yawe\\n\\nK. granule, grain (for example of salt)\\nL. ear of corn [corn cob]\\nM. juniper leaves\\nN. salt water\\nO. blood clots\\nP. field of grass\\nQ. blood\\nR. juniper branch\\nS. salt\\n\\nTranslate into Zuni:\\nwater\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}