{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1ac57704-aa20-5a76-820c-243ab2e108cc","task_key":"bbeh~5fnycc--full--1ac57704-aa20-5a76-820c-243ab2e108cc","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: An angel is blindfolded and holding a bat, preparing to hit a piñata shaped like Earth. Other angels are standing around watching.\\n\\n  Which of the following captions is the funniest?\\n(A) What's the point? It's probably all melted inside anyway.\\n(B) I hate when these things just have humans inside.\\n(C) But what is the string attached to?\\n(D) My favorite angel day piñata: this one has Milky Ways in it!\\n(E) Not to worry.... it's all plastic these days.\\n(F) It's harder than it looks. But after the last really good hit, all the dinosaurs flew off.\\n(G) It’s okay, all the good people were already Raptured.\\n(H) When are we going to tell him it's just magma inside?\\n(I) Not to worry.....he has trouble with a curve.\\n(J) I hope it's not going to spray hot magma everywhere like the last one.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}