{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a63fcf03-84d5-5d7e-b106-ccd5243dac82","task_key":"bbeh~5fnycc--full--a63fcf03-84d5-5d7e-b106-ccd5243dac82","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A therapist is sitting in a chair, listening to a broccoli character lying on a couch as if it were a patient.\\n\\n  Which of the following captions is the funniest?\\n(A) OK, I agree that everybody hates you, but is that necessarily a bad thing?\\n(B) So tell me, what has you so steamed?\\n(C) Now, if we could just get to the root of your problem.\\n(D) Don't take dog conduct personally.\\n(E) Tell me more about why you think people hate you.\\n(F) You were born a tree but identify as a woman?\\n(G) I know you don't like it when people say you look like a tree. Just be thankful you're not a banana.\\n(H) A lot of us are feeling uprooted these days.\\n(I) I'm afraid that you're in a persistent vegetative state.\\n(J) I have nightmares that I’ll be losing my hair in the fall!\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}