{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3d2ea9d0-aec8-5ce8-88f6-d0b4b2144c27","task_key":"bbeh~5fnycc--full--3d2ea9d0-aec8-5ce8-88f6-d0b4b2144c27","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man with an ice cream cone for a head is lying on a therapist's couch, talking to a therapist who is taking notes.\\n\\n  Which of the following captions is the funniest?\\n(A) Since I lost my sprinkles, life is just plain vanilla.\\n(B) “My wife complains that I’m too vanilla.”\\n(C) Is global warming truly an existential threat, or am I'm just going soft in the head?\\n(D) I live in fear of another meltdown\\n(E) Then I wake up and the sprinkles are gone.,\\n(F) She loves to lick my head up here and not down where all the good stuff is…. and it weirdly turns me on, is that normal?\\n(G) But everyone said it would be easy to find my Dairy Queen!\\n(H) The money and recognition are nice and all, but sometimes I just want to be plain old '\\\"Larry\\\" again.\\n(I) I really am trying to be more assertive, doctor, but everyone says I am a hopeless softie.\\n(J) Everyone I know tells me I'm cold.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}