{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3540b223-0a6b-5b64-be4a-dc39badace42","task_key":"bbeh~5fnycc--full--3540b223-0a6b-5b64-be4a-dc39badace42","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A real estate agent is showing a room to two potential buyers. Through the window, you can see outer space with planets and stars.\\n\\n  Which of the following captions is the funniest?\\n(A) Wait till you see the rooftop sun deck.\\n(B) The air's a little thin, but you'll love the view.\\n(C) . . . And you never hear the neighbors.\\n(D) You work remote, right?\\n(E) And just wait until you see the basement.\\n(F) You may be disappointed by the small size, but the view is out of this world.\\n(G) The atmosphere isn't much, but wait until you see the view.\\n(H) The value is going to be through the roof when they make Pluto a planet again.\\n(I) It’s a little far from the city, but the view is terrific.\\n(J) And it has Zoom background windows, this is one of my favorites...\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}