{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7d240ca1-cc1f-5420-a8d7-338d38aaafb3","task_key":"bbeh~5fnycc--full--7d240ca1-cc1f-5420-a8d7-338d38aaafb3","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man and a woman are in a swimming pool, playing with a beach ball. A cat is sitting on top of the beach ball.\\n\\n  Which of the following captions is the funniest?\\n(A) Get ready for the dunk shot!\\n(B) Dr. Kline said Whiskers’ anxiety should subside if we can get him to face his fears.\\n(C) I’m still ignoring you\\n(D) Mittens was beginning to reevaluate her goals as a social media influencer.\\n(E) I thought this might end his indifference.\\n(F) OK, it sounded like fun at the time!\\n(G) This game could be going a lot worse: my other pet is a porcupine.\\n(H) Now watch what happens when I spike it!\\n(I) Wow! This wine is good. We levitated the cat!\\n(J) I used to have nine normal lives before they made my Instagram channel\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}