{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64e53c27-1bd5-58f7-a774-93184408b629","task_key":"bbeh~5fnycc--full--64e53c27-1bd5-58f7-a774-93184408b629","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A group of people are trying to feed a giant dog that is emerging from a large structure. One person is on a ladder, offering food to the dog, while others are carrying large, colorful bones.\\n\\n  Which of the following captions is the funniest?\\n(A) Nick! Nack! Paddy! Wack! You know what to do!\\n(B) Okay, now assume belly rub formation!\\n(C) Yes, I mean literally throw me a bone.\\n(D) Hurry, while he's being such a good boy!\\n(E) Remember, he eats the green one last.\\n(F) Go fetch backup!\\n(G) Start up the Trojan Squirrel!\\n(H) Ok lads, I think we're ready to try \\\"shake a paw\\\" !\\n(I) We need more kibble- pass it on- no more bones- only more kibble!\\n(J) Who said these bones help his breath? I’m dying up here!\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}