{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b13a5cc5-ab63-5b96-b8d2-15ee551a23be","task_key":"bbeh~5fnycc--full--b13a5cc5-ab63-5b96-b8d2-15ee551a23be","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man in a military uniform is holding a glass of wine and talking to a woman in a black dress at a social event. The man has a long list of medals and ribbons on his uniform.\\n\\n  Which of the following captions is the funniest?\\n(A) That one is for my service at CVS.\\n(B) First, I was wined and dined. Now I'm running a tab. Firing by Tweet is only a matter of time.\\n(C) I see you served three tours in the Amazon fulfillment center.\\n(D) Tweets, for the latest directives.\\n(E) His list of insults grows by the tweet.\\n(F) Those are the legal disclaimers that go with all of my pins.\\n(G) It's his Tweet readout. You never know.\\n(H) Hopefully soon, China will start making our medals again.\\n(I) Black Friday Sale, 1993.\\n(J) Like everyone else here I'm for sale. Take a number.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}