{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a87121b1-41cd-50c5-87b5-822de949947e","task_key":"bbeh~5fnycc--full--a87121b1-41cd-50c5-87b5-822de949947e","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Three dinosaurs are playing musical instruments in a prehistoric landscape. One is playing a guitar, another is on drums, and the third is playing a keyboard. A meteor is seen falling from the sky in the background.\\n\\n  Which of the following captions is the funniest?\\n(A) That's the last time we cover ABBA\\n(B) So much for predictions of meteoric rise!\\n(C) Face it. It’ll be millions of years before we’re discovered.\\n(D) Rock 'n' Roll is Dead they say.\\n(E) Did you order special effects?\\n(F) I just feel like the hook on this one should go Goodness Gracious, something... something... something...\\n(G) And finally,we play  \\\"It's Now or Never\\\"\\n(H) I think we should go straight to the encore.\\n(I) 'Goodness gracious' ... 'goodness gracious' ... .Don't worry, it'll come to me.\\n(J) For those about to experience mass extinction, we salute you.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}