{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"213324ef-79f8-5f25-965a-0752d42cb1bc","task_key":"bbeh~5fnycc--full--213324ef-79f8-5f25-965a-0752d42cb1bc","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A group of people are at a social gathering, and one person is talking to a large inflatable tube man.\\n\\n  Which of the following captions is the funniest?\\n(A) Do you have to pee or something?\\n(B) I guess I don't have to ask what you do.\\n(C) I can't help but feel like you're trying to sell me something.\\n(D) I thought you said you have two left feet.\\n(E) You aren't the only giant windbag here.\\n(F) Don't you work at the used car dealership downtown?\\n(G) When you've got it, flaunt it!\\n(H) What do you say we blow this party?\\n(I) I apologize as I have difficulty recognizing faces out of context.\\n(J) So glad we can finally get together outside of the office\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}