{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0d572355-0515-51b7-afb2-d60f00b2ed12","task_key":"bbeh~5fnycc--full--0d572355-0515-51b7-afb2-d60f00b2ed12","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two cowboys on horseback are looking at a broken espresso machine and scattered coffee-making tools near a campfire in the desert.\\n\\n  Which of the following captions is the funniest?\\n(A) This must be the dude ranch, but where's the dude?\\n(B) What happened to the good old days?\\n(C) Looks like plain lattes... They must've been in some hurry.\\n(D) Smells like decaf. They can’t have gone far.\\n(E) I'd say we're trackin' a couple of city slickers.\\n(F) They say this, here, is the spot the great Milk Product Standoff occurred. Oat, soy, almond, cow--many a dander was raised, sighs heaved, eyes rolled. The eternal flame is in remembrance of  the baristas who had to put up with that crap that day.\\\"\\n(G) Well, I've always heard he's the fastest barista in the West.\\n(H) Yes, we just missed ‘em.... the Pony Espresso Gang.\\n(I) That Vente Latte still looks warm\\n(J) Fire's fresh, but with that much caffeine in 'em they could be miles away.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}