{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b748615c-4ec7-5ec6-b322-d380de9c2f57","task_key":"bbeh~5fnycc--full--b748615c-4ec7-5ec6-b322-d380de9c2f57","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A person is sitting at a restaurant table, holding an enormous menu that is much larger than they are. The waiter is standing next to the table, taking the order.\\n\\n  Which of the following captions is the funniest?\\n(A) The jumbo shrimp...excellent choice.\\n(B) What wouldn't you recommend?\\n(C) Hello, my name is Leo, and I'll be your speed reader this evening.\\n(D) Can I tell you about a few items that aren't on the menu?\\n(E) It makes the check appear not nearly so big.\\n(F) That IS the large-print edition, sir. Perhaps you would prefer the Braille.\\n(G) Are you ready or do you need a few more hours?\\n(H) Very good sir. The forklift with the wine list will be here presently.\\n(I) Just so you know, we are out of seventeen of the soups.\\n(J) Are you ready to order now or would you like a few more hours to decide?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}