{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17d126cc-12b1-5019-ba62-a8c7d41e3ba8","task_key":"bbeh~5fnycc--full--17d126cc-12b1-5019-ba62-a8c7d41e3ba8","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is sitting on a large egg in a living room while two people on a couch look at him in surprise.\\n\\n  Which of the following captions is the funniest?\\n(A) You think that’s hard? Let me tell you about laying it.\\n(B) Which came first?\\n(C) Can you take a turn, I have to go to the bathroom.\\n(D) At least the hard part's over.\\n(E) I told him not to fuck that brontosaur.\\n(F) At least the egg got laid.\\n(G) You did tell him to find an egg donor...\\n(H) Ted hasn't adjusted well to empty nest syndrome.\\n(I) I told him to get off his high horse and he came back with this.\\n(J) He's always hatching bird-brained ideas.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}