{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a352eda-4bc3-51f6-be0e-9230ef96ef15","task_key":"bbeh~5fnycc--full--7a352eda-4bc3-51f6-be0e-9230ef96ef15","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two angels are playing pool on a billiards table in the clouds. The balls on the table are planets, and a bearded figure, possibly representing God, is watching them.\\n\\n  Which of the following captions is the funniest?\\n(A) What's next in your downward spiral? A Taylor Swift concert?\\n(B) How many times must I tell you?  We do not roll dice with the universe, and we do not shoot pool with the solar system!\\n(C) Wings down! One foot on the clouds.\\n(D) If Galileo's up here, he needs to see this !\\n(E) Come on, this is serious....\\nFeet on the floor....No fly assists!\\n(F) Of course God doesn’t “play dice with the universe”, he plays pool\\nwith the universe!\\n(G) Thou shalt not play nine-ball without Pluto!\\n(H) Well, I’m glad someone finally found a use for those things!\\n(I) Let's see you do Earth in the side pocket..\\n(J) Where on Earth are you aiming\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}