{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cb3e34af-a8e4-5a76-ae7e-7c7243cdf941","task_key":"bbeh~5fnycc--full--cb3e34af-a8e4-5a76-ae7e-7c7243cdf941","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is sitting at a restaurant table with a bowl of soup in front of him. A tornado is coming out of the soup, and the waiter is standing nearby.\\n\\n  Which of the following captions is the funniest?\\n(A) How does that sound anything like 'tomato'?\\n(B) It spins the opposite way in Australia.\\n(C) I'm not a scientist, but I think there's a weather event in my soup.\\n(D) Do you recommend getting under the table or waiting it out?\\n(E) \\\"This is why restaurants in trailer parks never succeed.\\\"\\n(F) It's clever, but it will never replace the chocolate fountain.\\n(G) That’s it?  The chef recommends sheltering in place?\\n(H) This is a mistake. I ordered the tempest in a teapot.\\n(I) Perhaps you’d prefer the funnel cake.\\n\\n(Please underline the word “cake”.)\\n(J) You brought me the wrong drink. I ordered a hurricane.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}