{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"20d58a6d-1c46-57e0-9bb1-417419308c58","task_key":"bbeh~5fnycc--full--20d58a6d-1c46-57e0-9bb1-417419308c58","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two giraffes are standing close together with their necks intertwined in a complex knot.\\n\\n  Which of the following captions is the funniest?\\n(A) I'm so glad that we're not turtles.\\n(B) We've gone round and round and still can't see eye to eye.\\n(C) That’ll be the last time we watch a tennis match together.\\n(D) Let's try couple's Pilates next time.\\n(E) This is why I’m not a hugger, Gerald.\\n(F) So students, this is what a double helix looks like.\\n(G) I don't understand neck ties.\\n(H) Let's take some time off to unwind\\n(I) At least we didn't get our braces stuck this time.\\n(J) Wait!  Who amI?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}