{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c89bb5cd-25f3-5213-b2a7-83751a1a7623","task_key":"bbeh~5fmovie~5frecommendation--full--c89bb5cd-25f3-5213-b2a7-83751a1a7623","task_revision_id":"1","upstream_id":"","short_description":"Which option has more similar movies in terms of whether a group of people will…","config":"bbeh_movie_recommendation","split":"full","body":"{\"input\":\"Which option has more similar movies in terms of whether a group of people will like them or not?\\nOptions:\\n(A) Pulp Fiction, Forrest Gump, The Shawshank Redemption, The Fugitive, Dances with Wolves\\n(B) The City of Women, Avengers Infinity War - Part I, The Shawshank Redemption, The Fugitive, The Man Who Knew Too Much\\n(C) Forrest Gump, Pulp Fiction, The City of Women, Avengers Infinity War - Part I, The Fugitive\\n(D) The Fugitive, Pulp Fiction, Avengers Infinity War - Part I, Forrest Gump, The Man Who Knew Too Much\\n(E) Avengers Infinity War - Part I, The City of Women, Dances with Wolves, The Man Who Knew Too Much, Pulp Fiction\\n(F) The Shawshank Redemption, Pulp Fiction, The Man Who Knew Too Much, The City of Women, Forrest Gump\\n(G) Forrest Gump, Dances with Wolves, Avengers Infinity War - Part I, The City of Women, The Man Who Knew Too Much\\n(H) The Fugitive, Forrest Gump, Pulp Fiction, The City of Women, The Shawshank Redemption\\n(I) The City of Women, Avengers Infinity War - Part I, The Man Who Knew Too Much, Pulp Fiction, The Fugitive\\n(J) Avengers Infinity War - Part I, Forrest Gump, Dances with Wolves, The Shawshank Redemption, Pulp Fiction\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}