{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b83f7057-1755-5f7c-b8c4-22baf3614cfe","task_key":"bbeh~5fnycc--full--b83f7057-1755-5f7c-b8c4-22baf3614cfe","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A child is riding a coin-operated horse ride, dressed as a historical figure with a hat and cape. The mother is standing nearby, holding grocery bags and looking at the child.\\n\\n  Which of the following captions is the funniest?\\n(A) Get off your high horse, we can't afford a ride now.\\n(B) Damn Republicans!\\n(C) You're too short for that ride.\\n(D) Nice, but it’s time for “blankie” to stay home.\\n(E) You can ride to Moscow next week.\\n(F) You have to start somewhere.\\n(G) First Lunch.\\n(H) How many coins do you have left wench ?\\n(I) C'mon. We have to pick up Josephine from daycare\\n(J) Ugh, like father like son.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}