{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d085c3c3-34dd-5cd0-9dd7-d9c6b9f8cba6","task_key":"bbeh~5fnycc--full--d085c3c3-34dd-5cd0-9dd7-d9c6b9f8cba6","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is sitting at a table in a restaurant, reading a menu. A large, wet bear is standing next to the table, dripping water onto the floor.\\n\\n  Which of the following captions is the funniest?\\n(A) I'll have the penguin.\\n(B) You said you wanted it fresh!\\n(C) Honestly, you don't bear much resemblance to your profile picture...\\n(D) I told you to stay in the car and I would bring you your doggie bag.\\n(E) Thanks for checking on the salmon. How's the lamb?\\n(F) Maybe a little less rare.\\n(G) Is the salmon fresh caught or farm raised?\\n(H) Ok, let's skip the salmon.\\n(I) Thanks for coming out, chef, but now I understand why the fish was undercooked.\\n(J) I believe table 3 ordered the salmon.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}