{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6a68b055-abab-5998-b118-5265f442f104","task_key":"bbeh~5fmovie~5frecommendation--full--6a68b055-abab-5998-b118-5265f442f104","task_revision_id":"1","upstream_id":"","short_description":"Which option has more similar movies in terms of whether a group of people will…","config":"bbeh_movie_recommendation","split":"full","body":"{\"input\":\"Which option has more similar movies in terms of whether a group of people will like them or not?\\nOptions:\\n(A) Raiders of the Lost Ark, Everything You Always Wanted to Know About Sex  But Were Afraid to Ask, Star Wars Episode VII - The Force Awakens, The Proposal, Pulp Fiction\\n(B) Ernest Rides Again, Raiders of the Lost Ark, Pulp Fiction, Inside Out, Forrest Gump\\n(C) Ernest Rides Again, Forrest Gump, Everything You Always Wanted to Know About Sex  But Were Afraid to Ask, Inside Out, Star Wars Episode VII - The Force Awakens\\n(D) Star Wars Episode VII - The Force Awakens, Pulp Fiction, Inside Out, Raiders of the Lost Ark, Forrest Gump\\n(E) Ernest Rides Again, Pulp Fiction, The Proposal, Everything You Always Wanted to Know About Sex  But Were Afraid to Ask, Forrest Gump\\n(F) Everything You Always Wanted to Know About Sex  But Were Afraid to Ask, Ernest Rides Again, Forrest Gump, Inside Out, Raiders of the Lost Ark\\n(G) Ernest Rides Again, Everything You Always Wanted to Know About Sex  But Were Afraid to Ask, Raiders of the Lost Ark, The Proposal, Pulp Fiction\\n(H) Forrest Gump, Star Wars Episode VII - The Force Awakens, The Proposal, Pulp Fiction, Ernest Rides Again\\n(I) Pulp Fiction, Forrest Gump, Inside Out, Star Wars Episode VII - The Force Awakens, Ernest Rides Again\\n(J) Raiders of the Lost Ark, Pulp Fiction, Ernest Rides Again, Forrest Gump, Star Wars Episode VII - The Force Awakens\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}