{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64db0ef4-6b7a-50be-98ba-39e1c5b2d2fb","task_key":"bbeh~5fnycc--full--64db0ef4-6b7a-50be-98ba-39e1c5b2d2fb","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is holding hands with a child and a dog. The child looks upset while the dog is standing on its hind legs and wearing clothes.\\n\\n  Which of the following captions is the funniest?\\n(A) Don't be mad, Billy. When daddy gets his old job back, he can send both of you to boarding school!\\n(B) I think you’re going to be good friends. You’re adopted, too.\\n(C) It's okay, Billy. I'm sure you'll be adopted next.\\n(D) You said you wanted a baby brother... You said you wanted a puppy... Win-win!\\n(E) Son,you have to be tolerant of our dog identifying as your brother.\\n(F) Because he's a lot more obedient than you.\\n(G) Say it, Conner. ‘Rigel is family, not some son-of-a-you-know-what.’\\n(H) You’ve outgrown it, let him wear it.\\n(I) Buddy bit you because you weren't respecting his personal space.\\n(J) Son, you're old enough to learn about the facts of life. You'll never grow a tail.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}