{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa22018e-9d96-5450-97c8-6db9e33d99bf","task_key":"bbeh~5fnycc--full--aa22018e-9d96-5450-97c8-6db9e33d99bf","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two cowboys are playing croquet in a western town. They are using mallets to hit the balls while a few other cowboys and a horse watch from the sidelines.\\n\\n  Which of the following captions is the funniest?\\n(A) Sometimes it is a split decision.\\n(B) I have yet to see anyone take down ol' Davey Croquet in a shootout.\\n(C) I'm tempted to shoot them both.\\n(D) You know what would make this better? Horses.\\n(E) It starts at high noon but it can take forever\\n(F) They have some balls to play like that!\\n(G) Fine. But you'll have to rename it the Croquet Corral\\n(H) ' who knew the new gun laws would be so much fun!\\\"\\n(I) I’m all for gun control but this is kind of lame.\\n(J) They're from France.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}