{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f70a4358-ca80-54e3-82e2-7ce9a4054f75","task_key":"bbeh~5fnycc--full--f70a4358-ca80-54e3-82e2-7ce9a4054f75","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A dragon and a princess are playing table tennis in a meadow. In the background, a knight on a horse is approaching with a lance.\\n\\n  Which of the following captions is the funniest?\\n(A) Mask or no mask, I’m asking to see his vaccine card.\\n(B) Funny how he never rescues you when you're winning.\\n(C) I thought it was just going to be the two of us.\\n(D) And I don't give a \\\"damsel\\\" who is coming to save you.\\n(E) Don't tell me you posted another 'Distress' Instagram!\\n(F) Oh great, here comes Sir Wheel the third\\n(G) Don't be so impatient, St. George! I'm winning.\\n(H) If that twit as much as breaks my concentration, I will crisp him!\\n(I) Who rides on a horse with his sword hanging loose like that? I hope he’s at least wearing a mask.\\n(J) Lord of the Pings\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}