{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c359477b-2b0d-58bf-a1fb-94236fa1f8e5","task_key":"bbeh~5fnycc--full--c359477b-2b0d-58bf-a1fb-94236fa1f8e5","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A couple is packing their suitcases in a bedroom. The woman is packing clothes, while the man is packing a large stone wheel and a club.\\n\\n  Which of the following captions is the funniest?\\n(A) You said no technology this weekend,right?\\n(B) Reject modernity and embrace tradition? I don't think they meant that, Paul...\\n(C) I’m reinventing the wheel\\n(D) It's a back-to-basics resort.\\n(E) I got it to work without going on the web.\\n(F) I'm thinking that we'll just need the basics.\\n(G) I'm not implying anything about your dad.\\n(H) Trust me, it's fine if we check our bags.\\n(I) And then Joe Rogan said...\\n(J) It's been so long since I invented these, I'd forgotten all about them.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}