{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"60643557-09d5-581d-bac4-9f64a70733d6","task_key":"bbeh~5fnycc--full--60643557-09d5-581d-bac4-9f64a70733d6","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two people are sitting at a table in a restaurant, having a conversation over glasses of wine. One of them is wearing a suit of armor.\\n\\n  Which of the following captions is the funniest?\\n(A) In vino veritas? Surely you joust.\\n(B) That's not what I meant by a nice night out and you know it.\\n(C) So, do you like horses?\\n(D) Yes, I wrote that in my profile but I didn’t mean it literally.\\n(E) Sorry. It’s laundry day.\\n(F) Younger men haven't panned out, so I’m trying the Middle Ages.\\n(G) Cougar seeks millennial” didn’t mean the year 1000\\n(H) Oh, really? You think men are under attack?\\n(I) So that’s your best suit?\\n(J) Frankly, you look much older than your profile photo\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}