{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cd6007e6-3dae-52a6-9476-2049fcddb2e0","task_key":"bbeh~5fnycc--full--cd6007e6-3dae-52a6-9476-2049fcddb2e0","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A large group of sheep, all standing with their arms crossed, are facing two wolves dressed in suits. The wolves appear to be having a conversation.\\n\\n  Which of the following captions is the funniest?\\n(A) They don't hate us because we're wolves; they hate us because we're lawyers.\\n(B) ...Heh, Heh...Heh...We'll market them as Wolfadoodles....\\n(C) And so I said “trust me” and they did.\\n(D) We probably should have ditched the suits for sheep's clothing.\\n(E) You shouldn't have worn the wool suit.\\n(F) We have to ditch these wool blends.\\n(G) What do you say we go elsewhere for dinner?\\n(H) Herman, for God’s sake, stop drooling!\\n(I) We may have reached the functional limits of gerrymandering.\\n(J) I told you we'd be the only ones wearing suits\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}