{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9b3269e5-ca57-5abc-89d5-14b5d13496ac","task_key":"bbeh~5fnycc--full--9b3269e5-ca57-5abc-89d5-14b5d13496ac","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is sitting in a chair reading a book. A snake is coming out of a mounted shield on the wall and wrapping around the chair.\\n\\n  Which of the following captions is the funniest?\\n(A) Feed your mind, or feed me, your choice.\\n(B) You know that thing about 'draining the swamp' was just a ruse, right?\\n(C) Remember me?  Africa?  1978?\\n(D) If you don't want animals in your house, don't leave a big hole in the wall.\\n(E) Is this magical realism too real for you, Carl?\\n(F) Would you be interested in the knowledge of good and evil ?\\n(G) ''Bet that's the last time you have absinthe before bedtime.''\\n(H) Experiencing reptile dysfunction?\\n(I) Kiss?\\n(J) It's one of those books you need to read with the lights on. Trust me.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}