{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1956132e-deb2-52de-8ea0-4e752fcaf516","task_key":"bbeh~5fnycc--full--1956132e-deb2-52de-8ea0-4e752fcaf516","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two people are standing on a hill, holding glasses of wine, and looking at the moon. The moon has a face and is winking at them.\\n\\n  Which of the following captions is the funniest?\\n(A) What's in this wine? The moon just winked at us!\\n(B) I've never seen this guy before in my life, Babe, I swear!\\n(C) Maybe it was everything about last year, but, so far, I don't trust 2021 just yet.\\n(D) Damn. he knows what I'm thinking\\n(E) Houston we have a pervert\\n(F) Wine helps me connect with nature.\\n(G) I don't know if he's flirting with me, or just joking around.\\n(H) Don’t be fooled. He has a dark side.\\n(I) The universe is so vast and flirty.\\n(J) Oh, I didn’t know we were going on THIS kind of trip.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}