{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"40d76274-5123-5216-bd89-eda13193ed55","task_key":"bbeh~5fnycc--full--40d76274-5123-5216-bd89-eda13193ed55","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is tied up and standing in a bathtub with water up to his ankles. Another man, who looks like a detective or a gangster, is standing outside the tub, smoking a cigarette and watching him.\\n\\n  Which of the following captions is the funniest?\\n(A) There’ve been budget cuts\\n(B) Stop fighting it kid, everyone eventually sings in the shower.\\n(C) I don’t think this is how waterboarding works.\\n(D) Is this apartment 1A, or 1B?\\n(E) Mr. Trump told us to make it look like a suicide.\\n(F) We’d be doing this at the docks if it weren’t for the 10 o’clock curfew.\\n(G) Alright, he’s clean\\n(H) OK, that's it.  You're gonna sleep with the bathtub scum.\\n(I) What happened to the sleeping with fishes part?\\n(J) Are you ready to come clean or do you need more water?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}