{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6afa9049-7a5a-55d2-a5a1-c4bba9399040","task_key":"bbeh~5flinguini--full--6afa9049-7a5a-55d2-a5a1-c4bba9399040","task_revision_id":"1","upstream_id":"","short_description":"Here are some verb forms in Plains Cree and their English translations:","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Here are some verb forms in Plains Cree and their English translations:\\n\\n1. nimawinēskomik | he challenges me\\n2. kiwāpamināwāw | you_{pl} see me\\n3. kinakininān | you_{sg} stop us, you_{pl} stop us\\n4. nikakwēcimānānak | we ask them\\n5. kiwīcihāw | you_{sg} help him\\n6. nikakwēcimānān | we ask him\\n7. niwāpamāw | I see him\\n8. kimawinēskomin | you{sg} challenge me\\n9. nimawinēskomikonānak | they challenge us\\n10. kikakwēcimik | he asks you_{sg}\\n11. kiwāpamitinān | we see you_{sg}, we see you_{pl}\\n12. ninakināwak | I stop them\\n13. kiwīcihikowāw | he helps you_{pl}\\n14. kinakinikwak | they stop you_{sg}\\n15. kiwāpamāwāwak | you_{pl} see them\\n16. kiwīcihitin | I help you_{sg}\\n\\nHere are some more verb forms in Plains Cree and their English translations:\\n\\n17. ē-nakināyēk | as you_{pl} stop him\\n18. ē-wīcihak | as I help him\\n19. mawinēskomikoyāhkwāwi | if they challenge us\\n20. wāpamaci | if you_{sg} see him\\n21. wīcihitako | if I help you_{pl}\\n\\nTranslate into Plains Cree:\\nif we ask them\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}