{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a38bd84b-cbfa-538d-a3fa-d6752080c255","task_key":"bbeh~5fnycc--full--a38bd84b-cbfa-538d-a3fa-d6752080c255","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: People are in an office lobby. Two individuals are standing inside a pair of cubicles that are designed to look like they are part of the wall.\\n\\n  Which of the following captions is the funniest?\\n(A) That must be one of those \\\"standing desks\\\" that everyone has been talking about.\\n(B) Ignore him. He just does it to get sympathy.\\n(C) He heated up fish in the microwave. Don't do that.\\n(D) It was either that for five lunch hours or a three hundred million dollar fine\\n(E) Boy the penalty for early withdrawal is pretty severe at this bank\\n(F) Some people would do anything to avoid lunch with the boss.\\n(G) I hear there is a new HR Manager.\\n(H) I thought they were putting art in the lobby, not Art in the lobby.\\n(I) He's the stock boy.\\n(J) Bob was nice to a customer, I hear.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}