{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a784a551-daca-55fb-b6db-fe21f3812dda","task_key":"bbeh~5fnycc--full--a784a551-daca-55fb-b6db-fe21f3812dda","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A businessman in a suit is sitting on a subway next to a caveman holding a club. They both appear to be looking forward.\\n\\n  Which of the following captions is the funniest?\\n(A) I can sense your judgment of my manspread.\\n(B) So it kind of started with a drum circle at this men's weekend...\\n(C) Are you SURE this will appeal to the jury?\\n(D) You know the CIA runs Facebook, right?\\n(E) I'm sensing some toxic masculinity.\\n(F) You have no idea how long I had to wait for this train.\\n(G) I'm so glad open carry is allowed here.\\n(H) If you stay on the Palio diet long enough, this will happen.\\n(I) I’m still having trouble with the concept of dressing for success.\\n(J) Why does everyone keep asking me if I’m related to Andrew Cuomo? I’m a caveman. Not an asshole.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}