{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d0eeae4-7c9d-5c61-964b-00a09c3140b6","task_key":"bbeh~5fnycc--full--9d0eeae4-7c9d-5c61-964b-00a09c3140b6","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A couple is sitting at a table in a train station, looking at menus. A waiter is standing next to them, taking their order. The station is busy with people waiting for trains.\\n\\n  Which of the following captions is the funniest?\\n(A) Let me guess, you don't like this table either?\\n(B) May I recommend absolutely nothing.Get out while you still can.\\n(C) I didn't think you could do worse than taking me to Subway for our anniversary. I was wrong.\\n(D) Will that be to-go, Sir?\\n(E) Sorry, we don't accept tokens or Metrocards.\\n(F) All of our specials are track-to-table and steamed on site.\\n(G) The special today can be served between the 6:24 and the 7:32.\\n(H) For the Express menu, you need to go down to 14th St.\\n(I) Might I recommend sitting somewhere else?\\n(J) We're known for our Tube Steak.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}