{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"14ab5ad2-3b6a-525f-8756-a4d983925fac","task_key":"bbeh~5fnycc--full--14ab5ad2-3b6a-525f-8756-a4d983925fac","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A person is lying on a couch in a therapist's office, talking to the therapist. There is a giant clock on the wall behind them.\\n\\n  Which of the following captions is the funniest?\\n(A) Now you can stop asking me what time it is\\n(B) When I snap my fingers you will awaken and forget the entire year of 2023.\\n(C) So you think you're good at picking up on subtle hints.  Maybe we should discuss that next time.\\n(D) I’m very big on tick-tok.\\n(E) Your mother never gave you enough of her time, did she?\\n(F) It was a gift from my six o'clock.\\n(G) You can still reach closure if you use your timeouts wisely.\\n(H) Ticking?  Actually I find your breathing more annoying.\\n(I) The good news, if the clock falls your troubles are over.\\n(J) Yeah, I told the interior designers  that I wish I had more time in my office to decorate\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}