{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"25c8acb5-80a0-5eb8-bc43-769e778211dd","task_key":"bbeh~5fnycc--full--25c8acb5-80a0-5eb8-bc43-769e778211dd","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two whales are relaxing on a beach. One whale is holding a drink with a straw, while the other is reading a book and wearing sunglasses. The city skyline is visible in the background.\\n\\n  Which of the following captions is the funniest?\\n(A) How Alaska has changed.  Not that long ago it was too cold to get out of the water; let alone get a tan!\\n(B) We'll make better time if we cut through the Panama Canal.\\n(C) I liked it when he said, \\\"You two don't belong in the cruise ship pool.\\\"\\n(D) Just think Mother,thanks to climate change we will not need to migrate to breed ever again.\\n(E) Now that humans are extinct, we can enjoy the beach again.\\n(F) Frank, are we suicidal or just chillin’\\n(G) And then he said, \\\"Shaken, or stirred?\\\"\\n(H) All bodies are beach bodies!\\n(I) Might as well enjoy this before it becomes uninhabitable\\n(J) Of course I turned off echolocation. I'm on vacation!\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}