{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"26dd3756-cab9-5e0a-ae82-039a71e51770","task_key":"bbeh~5fnycc--full--26dd3756-cab9-5e0a-ae82-039a71e51770","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A bartender is pouring a drink for a customer who appears to be Humpty Dumpty. Humpty Dumpty is sitting at the bar with bandages and cracks all over his shell, indicating he has been put back together.\\n\\n  Which of the following captions is the funniest?\\n(A) Well, I lost a little albumen, but my yolk's still intact.\\n(B) I feel like hell in an Easter basket.\\n(C) Day Trader on Wall Street ... it was my dream job.\\n(D) Just don't let me get pickled.\\n(E) If the King's horses call, I'm not here.\\n(F) Make it a double, I might as well fall off the wagon too.\\n(G) I mean, who sends horses?\\n(H) Thank God it was just a bar stool, Jimmy.\\n(I) It was a mess. About a dozen of us were just sitting on a wall. But when a fight started, the rest of them dumped me and scrambled.\\n(J) They put me together again, but it’ll cost me an arm and a leg.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}