{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d7f1341-b62d-5d1b-a8ac-e19b2f3ddcca","task_key":"bbeh~5fnycc--full--9d7f1341-b62d-5d1b-a8ac-e19b2f3ddcca","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man in a business suit is being thrown off a bucking horse. The horse is in mid-air, and the man is flying through the air with a surprised expression.\\n\\n  Which of the following captions is the funniest?\\n(A) Just tell him the buck stops here!\\n(B) You won't believe the new app I got for only a buck!\\n(C) Fine.  Next time I'll take Uber!\\n(D) Test driving a new mustang, what are you up to?\\n(E) You were right, honey! I did need to get out of my comfort zone.\\n(F) Gotta go, I'll text you when I land.\\n(G) Hi honey,can you call our insurance company and see if they cover stupid.\\n(H) Oh, I'm absolutely hands free right now.\\n(I) I'll be on the ground soon\\n(J) Obviously, this is my first rodeo.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}