{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"85d1a1e7-4818-5e47-92d7-521d267a2537","task_key":"bbeh~5flinguini--full--85d1a1e7-4818-5e47-92d7-521d267a2537","task_revision_id":"1","upstream_id":"","short_description":"Here are some sentences in Coastal Marind and their English translations:","config":"bbeh_linguini","split":"full","body":"{\"input\":\"Here are some sentences in Coastal Marind and their English translations:\\n\\n1. ob-dahetok | You_{sg} really returned.\\n2. nake-dahetok | We returned.\\n3. nambat-ihwin | Aw, I cried.\\n4. nambe-otab | We really chased you_{pl} away.\\n5. um-kulaɣawn | He laughed at you_{sg} in vain.\\n6. e-keseh | You_{pl} spat on him.\\n7. ubate-huɣanab | Aw, you_{sg} silenced us.\\n8. nakum-hoɣab | I silenced you_{pl} in vain.\\n9. nambame-ɣadaɣawn | Oh, we left you_{sg} behind!\\n10. ebe-ɣadanawn | You_{pl} really left us behind.\\n11. ibatum-ɣadewn | Aw, you_{pl} left him behind in vain.\\n12. a-isanab | He cut me in half.\\n13. obam-eseb | Oh, you_{sg} cut him in half!\\n\\nThe following verbs have a grammatical difference from the verbs above:\\n\\n14. bate-tamanab | Aw, we floated to the surface.\\n15. ba-tamab | You_{pl} really floated to the surface.\\n16. a-ɣuyaɣah | You_{sg} shuddered.\\n17. bam-ɣuyanah | Oh, I shuddered!\\n\\nTranslate into Coastal Marind:\\nYou_{pl} really spat on us in vain.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}