{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"43670bee-b3c0-5b9b-8cb5-b3c257d4c599","task_key":"bbeh~5fnycc--full--43670bee-b3c0-5b9b-8cb5-b3c257d4c599","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two castaways are on a deserted island with a grand piano. One is reading sheet music while the other sits at the piano. There are bottles and more sheet music scattered around them.\\n\\n  Which of the following captions is the funniest?\\n(A) Damn...They're all from Jim Croce!\\n(B) Another noise complaint?!\\n(C) It says the first five bottle messages were free, but if we want the rest of the composition we need a paid subscription.\\n(D) …take one down, pass it around, 96 bottles of beer on the wall\\n(E) It’s the score to ‘Come Sail Away’. So not helpful.\\n(F) They all want you to stop playing Barry Manilow songs.\\n(G) Ok . This one’s from Sting.\\n(H) Looks like the warranty doesn't cover shipwrecks, or sand, or water, and the closest technician is 4458 miles away.\\n(I) I know, I know, but at least there is a no-mask policy here.\\n(J) They are having a party for Gilligan, and want to know if we can make it\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}