{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae0f1e0e-3ce1-5196-a3e4-114015b2d512","task_key":"bbeh~5fnycc--full--ae0f1e0e-3ce1-5196-a3e4-114015b2d512","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is holding a lamp, and a genie in a business suit is emerging from it. The genie appears to be giving advice or instructions.\\n\\n  Which of the following captions is the funniest?\\n(A) keep this off the books, ok!?\\n(B) Your third and last investment wish, buy Bitcoin…😂…sorry, bad joke.\\n(C) Won’t they get us for insider trading?\\n(D) . . . and aforementioned client, viz 'The Lamp,' wishes to state, reiterate, and confirm that said rub was nonconsensual.\\n(E) I will grant you three deductions on your taxes.\\n(F) You get three wishes, but I can’t get you another extension on your ‘22 return.\\n(G) Sorry Donald but there are limits to what even I  can do!\\n(H) Word peace? What are you, some kind of nut?\\n(I) Yeah, well, you don't look like Aladdin either\\n(J) I hereby grant you 3 tax deductions.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}