{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5b5aaaf6-d813-536d-8c34-446b118713fd","task_key":"bbeh~5fnycc--full--5b5aaaf6-d813-536d-8c34-446b118713fd","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A bartender is serving drinks to three unusual customers: a rock, a piece of paper, and a pair of scissors. Each of the objects has a face and appears to be engaged in conversation with the bartender.\\n\\n  Which of the following captions is the funniest?\\n(A) That’s it fellas… no more games.\\n(B) And who's covering the tab?\\n(C) Here, we draw straws.\\n(D) Last time you guys were here, Rock got Scissors smashed and Paper failed to cover the bill.\\n(E) Alright already, one of you just order.\\n(F) Alright, one of you has to decide.\\n(G) Three old fashioned, coming right up.\\n(H) Just flip a coin.\\n(I) Lemme guess... Shots?\\n(J) What'll it not be?\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}