{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7ae5cf8d-9a8a-501b-838a-d94c21e2c729","task_key":"bbeh~5fnycc--full--7ae5cf8d-9a8a-501b-838a-d94c21e2c729","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A man is having a yard sale in front of his house. Among the items for sale is a large submarine parked on the lawn. A potential buyer is talking to the man in a naval uniform who is running the sale.\\n\\n  Which of the following captions is the funniest?\\n(A) Your tax dollars already paid for it but I will have to charge you a moving fee.\\n(B) That, my friend, is a nuclear powered pool toy.\\n(C) The paperwork's in the glove box. Title, owner's manual, nuclear launch codes. All there.\\n(D) It was only driven once, by a little old admiral from Virginia.\\n(E) Australia wouldn't take it. It's yours for nickel.\\n(F) I’m only looking to sublease, will that be a problem?\\n(G) Is it still on warranty?\\n(H) We decided it was time to clean out the basement.\\n(I) Last question, Would you be willing to throw in the skis?\\n(J) It’s roomy, yet discreet, once you have it down in the water.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}