{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5237c975-b79f-55e4-a5e4-53ed0e59143d","task_key":"bbeh~5fnycc--full--5237c975-b79f-55e4-a5e4-53ed0e59143d","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Three cavemen are standing inside a cave, looking at a small dinosaur that is lying on the ground. One of the cavemen appears to be talking to the others.\\n\\n  Which of the following captions is the funniest?\\n(A) It's ok he's friendly.\\n(B) Afraid I’m currently detained\\n(C) He’s developed an attachment.\\n(D) He hasn't evolved to sniffing yet.\\n(E) Thought it would be cute to teach it some idioms, but maybe I shouldn't have started with \\\"break a leg\\\".\\n(F) They’re known for their unconditional love.\\n(G) The evolution of depth perception took a major step forward today.\\n(H) I would absolutely love to go out to dinner as we planned, but I've just invented the zero, discovered fire, and perfected the axle, and now THIS!\\n(I) I got house arrest with an ankle monitor.\\n(J) It's a little painful, but I'm teaching him to heel.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}