{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f5673aa7-2ca6-51a1-b9b8-7a22220accc0","task_key":"bbeh~5fnycc--full--f5673aa7-2ca6-51a1-b9b8-7a22220accc0","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A scientist in a lab coat is standing in a maze, facing a giant mouse. The mouse is much larger than the scientist, who looks surprised.\\n\\n  Which of the following captions is the funniest?\\n(A) So you promise!  I find the exit, and you'll be there waiting for me.\\n(B) You win.  Go home.\\n(C) Can you tell me where the cheese is?\\n(D) Payment for the experiment will be waiting for you at the exit.\\n(E) I agree, lately this research feels like nothing but dead ends.\\n(F) Good mouse, but no more treats for you.\\n(G) After the human trials, I'm not so sure which one of us is more genetically modified.\\n(H) No, this is The Labyrinth.  The Maze is on the 4th floor.\\n(I) Well, this side effect was unexpected.\\n(J) That concludes our study. You can see yourself out.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}