{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4ecf105a-2cf8-5e8e-a530-daf63ab50681","task_key":"bbeh~5flinguini--full--4ecf105a-2cf8-5e8e-a530-daf63ab50681","task_revision_id":"1","upstream_id":"","short_description":"Here are some words and word combinations in Abui and their English translations…","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Here are some words and word combinations in Abui and their English translations in arbitrary order:\\n\\n1. abang\\n2. atáng heya\\n3. bataa hawata\\n4. dekafi\\n5. ebataa hatáng\\n6. ekuda hawata\\n7. falepak hawei\\n8. hatáng hamin\\n9. helui\\n10. maama hefalepak\\n11. napong\\n12. rièng\\n13. ritama\\n14. riya hatáng\\n15. tama habang\\n16. tamin\\n17. tefe hawei\\n\\na. his fingertip\\nb. your (sg.) branch\\nc. my face\\nd. one’s own rope\\ne. your (sg.) shoulder\\nf. your (pl.) mother’s hand\\ng. our pigs’ ears\\n(the ear of the pig of each of us)\\nh. father’s pistol\\ni. your (sg.) horse’s neck\\nj. trigger\\nk. your (pl.) eyes\\nl. our noses\\n(the nose of each of us)\\nm. his knife\\nn. seashore\\no. upper part of a tree\\np. your (sg.) thumb\\nq. your (pl.) sea\\n\\nTranslate into Abui:\\nyour (pl.) knife\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}