{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"00502bea-4ed9-57e0-9255-c91d047c3d75","task_key":"bbeh~5fnycc--full--00502bea-4ed9-57e0-9255-c91d047c3d75","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A doctor is examining a patient who looks like a giant potato with human features, including a mustache, eyes, and a hat. The patient is sitting on an examination table.\\n\\n  Which of the following captions is the funniest?\\n(A) You have  a large tuberous growth, located, well, I guess you'd say, all over\\n(B) They're all interchangeable, really. We could, say, put your ear where your nose is...\\n(C) Only two eyes but otherwise perfectly healthy.\\n(D) The procedure went excellent Mr. Head, all your parts are back in place.\\n(E) So you lost your right arm?\\n(F) Seriously, you need to get off the couch once in a while.\\n(G) I’m sorry sir, but the problem appears to be entirely in your head.\\n(H) What is your level of pain? Six-potato, seven-potato, eight-potato, more?\\n(I) Actually, you are in pretty good shape for your age, I have never seen a potato with just two eyes!\\n(J) As I told you at your last exam, your high potassium levels\\n are hereditary.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}