{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"98aa9b2f-9438-57a4-b99c-0dbc6eba376e","task_key":"bbeh~5fnycc--full--98aa9b2f-9438-57a4-b99c-0dbc6eba376e","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two people are sitting outside in chairs. One of them is being attacked by a giant mosquito while the other looks on in shock.\\n\\n  Which of the following captions is the funniest?\\n(A) Yup this one right here, she’s the one who squished Carol... left Dean to take care of 2000 babies on his own.\\n(B) I'll call the cops, but what should I tell 'em?\\n(C) I told you they’re immune to citronella.\\n(D) Mosquitoes have never done that to me before\\n(E) Well, Linda, it looks like you won the disaster prediction pool, but I’d say it’s something of a bittersweet victory\\n(F) Remember: Tap don’t scratch\\n(G) See? I told you. They just don’t bother me.\\n(H) Oh good, it is just Zika.\\n(I) In 2019, I would have been both confused and alarmed by this\\n(J) No, no, I'm sure I ordered \\\"two giant MOJITOS.\\\"\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}