{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8e14df82-cd52-510b-8b74-ea1d6ae57ce3","task_key":"bbeh~5fnycc--full--8e14df82-cd52-510b-8b74-ea1d6ae57ce3","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A woman is sitting on an airplane, looking surprised at the giant ant sitting next to her, which is reading a book.\\n\\n  Which of the following captions is the funniest?\\n(A) Memo to self: no more Microdosing\\n(B) I guess those reports about the emergence of a more sophisticated strain of flying ant are true…\\n(C) And to think I paid a premium for the extra legroom.\\n(D) I hope they don't get our meals mixed up!\\n(E) If you're organizing a picnic at  feet, can I join? \\n(F) I'm off to visit my great aunt.  What about you?\\n(G) Climate change.\\n(H) So people really look like ants from up here, right?\\n(I) She’s probably going to want to talk to me, the whole flight. What a pest!\\n(J) Kafka, you're awfully calm for having emptied the plane that fast.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}