{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fca60d59-b850-5327-af20-19d3293598f1","task_key":"bbeh~5fnycc--full--fca60d59-b850-5327-af20-19d3293598f1","task_revision_id":"1","upstream_id":"","short_description":"The following is a description of a funny cartoon for the New Yorker Caption…","config":"bbeh_nycc","split":"full","body":"{\"input\":\"The following is a description of a funny cartoon for the New Yorker Caption Contest\\n\\n  Description: A courtroom scene where a teddy bear is sitting in the witness stand. A lawyer is standing nearby, and the judge is seated at the bench.\\n\\n  Which of the following captions is the funniest?\\n(A) Your Honor, when I said 'bear witness', this is not what I meant\\n(B) Your Honor, my client was at the scene of the crime but did not personally steal any cookies.\\n(C) Objection sustained. That's not what \\\"bear witness\\\" means.\\n(D) How long had you been sleeping with the victim?\\n(E) The witness will require an interpreter, Your Honor.\\n(F) Your Honor, I’d like to hold this witness for further questioning.\\n(G) Where were you on the night of December 24th?\\n(H) I want to treat him as a hostile witness, but geez.\\n(I) Are you or are you NOT made of sustainably sourced materials???\\n(J) He threatened to beat the stuffing out of me!\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}