{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"340ca940-7c23-5216-ba4b-6624e58a43ea","task_key":"bbeh~5fnycc--full--340ca940-7c23-5216-ba4b-6624e58a43ea","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is sitting on an examination table in a doctor's office. He is wrapped up in a large snake, and the doctor is standing in front of him, holding a clipboard and talking to him.\\n\\n  Which of the following captions is the funniest?\\n(A) Looks like a reptile dysfunction\\n(B) How was your vacation?\\n(C) Sorry, but the insurance will not approve the operation without your signature.\\n(D) Of course an MRI will be more conclusive.\\n(E) What exactly is the problem?\\n(F) We'll need to get some bloodwork done just to be sure.\\n(G) Any shortness of breath?\\n(H) No,she seems pretty healthy, but you were wise to bring her in regardless.\\n(I) The good news is, it’s not a tumor”\\n(J) On a scale of hug to asphyxiation, how would you describe this tightness in your chest?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}