{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"667d26cb-f77f-5fd4-96c7-7037a909c27b","task_key":"bbeh~5fnycc--full--667d26cb-f77f-5fd4-96c7-7037a909c27b","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is running inside a giant hamster wheel in an office-like room. The door to the room is open, and there is a window on the left side.\\n\\n  Which of the following captions is the funniest?\\n(A) Funny, unfunny, unfunny, somewhat funny, . . . .\\n(B) I don't mind powering the building, but I'm gonna need the chips changed more often\\n(C) How the Secret Service in the Trump Hotel both stays in shape and supplies the cheap energy to cool the Yuge Bannon Beer Cave in the penthouse.\\n(D) I'll get back to you later — gotta run.\\n(E) I can't believe it, I got such a great deal - $80 a month, and they have showers too!\\n(F) Quick! Someone close the door before the cat gets in.\\n(G) No wonder this gym was so cheap.\\n(H) Well, yeah, of course I'm going somewhere here at Headquarters. What kind of question is that??!\\n(I) At least it's air conditioned.\\n(J) Not much. Just spinning my wheel. And, you?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}