{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64d4e122-f217-5acd-a5bf-5a27e77efc2c","task_key":"bbeh~5fnycc--full--64d4e122-f217-5acd-a5bf-5a27e77efc2c","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: Two men are standing in an elevator wearing life preservers. Another man is standing outside the elevator looking at them.\\n\\n  Which of the following captions is the funniest?\\n(A) Sorry.  No roompah.\\n(B) What's better than one sousaphone in the stairwell?\\n(C) My dear man, have you by chance come across an orchestra with adequate mid-range and highs but a rather feeble low end?\\n(D) Going up will be a big blast !\\n(E) Shelter in place!?  Fine, but we're still having this parade!\\n(F) Care to accompany us?\\n(G) Sure there's room, but not with that big 'man purse'.\\n(H) What are you waiting for?  You got something against elevator music?\\n(I) Neither one of us plays, but we find they keep people at least six feet away.\\n(J) This is a standing-room-only performance.\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}