{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"162a16fa-7610-59cc-b5e6-01dfc5c71b80","task_key":"bbeh~5fnycc--full--162a16fa-7610-59cc-b5e6-01dfc5c71b80","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man and a woman are standing in a kitchen that is heavily secured. The refrigerator and cabinets are chained and padlocked, and there are bear traps and spikes on the floor.\\n\\n  Which of the following captions is the funniest?\\n(A) And the bacon still got out!\\n(B) Yes. Or we could try willpower.\\\"\\n(C) Sweetie, I love your art. I do. But this is unbearable.\\n(D) Somehow the calories still escape and find me.\\n(E) I think you’re going a bit overboard. It’s only a mouse.\\n(F) No.... I don’t think it’s an over-reaction.\\n(G) Did you remember to get your prescription out?\\n(H) I don't know about you, but I'm thinking takeout\\n(I) Ben and Jerry are in that freezer, Babe.\\nI’m goin’ in.\\n(J) Think of how few dishes we’ll have to do.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}