{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1500325c-98e8-5df2-96ba-2b8b140f851f","task_key":"bbeh~5fnycc--full--1500325c-98e8-5df2-96ba-2b8b140f851f","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is standing in a room, talking on the phone. He has a knife stuck in his back with a piece of paper attached to it.\\n\\n  Which of the following captions is the funniest?\\n(A) It's for me but I can't read it.\\n(B) Actually, I would like to shoot the messenger.\\n(C) All I remember was that he looked like a monk\\n(D) These manuscript rejections are killing me.\\n(E) So much for \\\"The Art of the Deal.\\\"\\n(F) What letter?\\n(G) At this moment it's a little difficult to read the fine print.\\n(H) I'm sure there's an emoticon that would have worked just as well.\\n(I) I'M CALLING TO COMPLAIN ABOUT HOW YOUR PAPER IS DELIVERED!!\\n(J) Hello, Roman Times? About your paper boy, Brutus...\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}