{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"475afad7-0b38-5a4f-bdec-f62c364f3414","task_key":"bbeh~5fnycc--full--475afad7-0b38-5a4f-bdec-f62c364f3414","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man behind a counter is looking at a sad clown standing outside the window of a place called \\\"Sam's.\\\"\\n\\n  Which of the following captions is the funniest?\\n(A) I know you're the only one performing but the inauguration is still a pretty big deal.\\n(B) Now before you start, I don't want any funny business\\n(C) No funny business\\n(D) If you can't find a way to juggle a career and family, what hope is there for the rest of us?\\n(E) Well, it suits you better than the president costume.\\n(F) Squirt me with that flower again and Sam is going to kick your ass!\\n(G) I suppose you're looking for a punchline.\\n(H) Sorry, no seltzer.\\n(I) Clown shirt, clown shoes, no service.\\n(J) We don't serve clowns here ... we just elect them to the highest office in the land.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}