{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9f16bf43-b050-5fbc-a3cc-d2140e53cfc6","task_key":"bbeh~5fnycc--full--9f16bf43-b050-5fbc-a3cc-d2140e53cfc6","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man and a woman are standing in a room filled with bowling balls and pins. The woman is holding a rolling pin, and the man is holding two bowling balls.\\n\\n  Which of the following captions is the funniest?\\n(A) So what if they were on sale?\\n(B) Define obsession.\\n(C) And still the cat got through?\\n(D) You gotta love Amazon Prime.\\n(E) Hmm. Why do I think our relationship is in the gutter?\\n(F) Remind me again why immersion bowling seemed like a good idea for us?\\n(G) I'm thinking of quitting at the bowling alley. Work from home has been awful.\\n(H) What do you mean the pins don’t social distance?\\n(I) Maybe joining the gym was a better idea after all\\n(J) It’s time for me to split.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}