{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"70ef0a9f-828d-5b10-848b-6ec412149671","task_key":"bbeh~5fnycc--full--70ef0a9f-828d-5b10-848b-6ec412149671","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is lying in a coffin underground, looking surprised. Next to him, a skeleton in another coffin is holding a bouquet of flowers.\\n\\n  Which of the following captions is the funniest?\\n(A) OK, I see you lost a lot of weight. You don't have to gloat.\\n(B) Not so soon pal, it's the same dream I have before my MRI!\\n(C) Mandy, no. I'm still your cousin.\\n(D) That’s sweet Susan, but I’m having a lot of trouble with all of this.\\n(E) No, I don't have a smoke. And no, I don't want to join your neighborhood association.\\n(F) Please!  I just got OUT of a relationship!\\n(G) I'm not nearly as happy to see you as you are to see me.\\n(H) Hey, newbie. I’m Alex. We’re going to be neighbors for eternity. Isn’t that great? Do you like yacht rock? Tuesdays are taco night. I’m a pescatarian. I see you got the lid off by yourself. I don’t know why they buried me naked. I love cribbage...\\n(I) Is this Heaven or Hell???\\n(J) Arguing about politics is what killed me, so please shut up!\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}