{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ed53999f-679f-50c8-b0be-5e029cf11097","task_key":"bbeh~5flinguini--full--ed53999f-679f-50c8-b0be-5e029cf11097","task_revision_id":"1","upstream_id":"","short_description":"Recent contact with Western civilization has brought some changes into the…","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Recent contact with Western civilization has brought some changes into the Iatmül language. New words and expressions have emerged and some existing ones have developed new meanings. Here are some words and word combinations in Iatmül and their English translations in arbitrary order:\\n\\n    1. guna vaala\\n2. ka’ik\\n3. ka’ikgu\\n4. klawun\\n5. laavu\\n6. laavuga vi’\\n7. laavuga\\n8. ni’bu\\n9. ni’buna vaala\\n10. nyaka’ik\\n11. vi\\n12. vi’wun\\n13. walini’bana bâk\\n14. walini’bana gu\\n15. walini’bana vi\\n\\na. banana\\nb. cow\\nc. canoe\\nd. book\\ne. carbonated beverage, alcohol\\nf. rifle\\ng. picture, shadow\\nh. car\\ni. I got\\nj. mirror\\nk. I saw\\nl. watch (timepiece)\\nm. spear\\nn. to read\\no. ground, land\\n\\nTranslate into Iatmül:\\nwhite people\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}