{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a68f0415-5df7-5f81-b8db-3150ee27e31d","task_key":"bbeh~5fnycc--full--a68f0415-5df7-5f81-b8db-3150ee27e31d","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A flight attendant dressed as a magician is performing a magic trick for passengers on an airplane. The passengers, including a child, are watching with interest.\\n\\n  Which of the following captions is the funniest?\\n(A) And now...nutless chow mix!!!\\\"\\n(B) And for my next trick, I’ll make everyone miss their connecting flight.\\n(C) Abracadabra!  You deserve first class and have been chosen to lead the revolt!\\n(D) For my next trick, I'll shrink your legroom even more!\\n(E) Imagine, life without the politics of Earth...we're off to Mars to start fresh and overcome our mistakes.\\n(F) Here at Spirit Airlines, you just have to BELIEVE in the magic of customer service and leg space\\n(G) Welcome to Oz!\\n(H) And now I'll need a volunteer for my vanishing act. This flight is overbooked.\\n(I) I'm here to enhance your flight experience.\\n(J) Now for the illusion of safety.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}