{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2eb1cb8b-a60b-5673-af45-247435af8630","task_key":"bbeh~5fnycc--full--2eb1cb8b-a60b-5673-af45-247435af8630","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A knight in armor and a lifeguard are sitting on lifeguard chairs on a beach. The knight is holding a spear, and there is a sandcastle between them.\\n\\n  Which of the following captions is the funniest?\\n(A) SPF 3000. It's  highly recommended.\\n(B) It''s only temporary.\\n(C) This time I’m not letting any wooden Seahorse in my castle!\\n(D) I’ll joust you for it.\\n(E) So I guess I misunderstood when I heard “night watch”…\\n(F) Are you even taking this seriously?\\n(G) We’re going to need a bigger moat!\\n(H) Yes, you could say I take my job seriously.\\n(I) There is a high serf advisory.\\n(J) All the agency said was the job involves castles and a coastline.  *You* try guarding the south approach from Visigoths in a Speedo.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}