{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dabdb05b-a599-578d-a3ff-6b45418f5825","task_key":"bbeh~5fweb~5fof~5flies--full--dabdb05b-a599-578d-a3ff-6b45418f5825","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Inga says only one of Jaymie, Ozzy and Marcus lies. Muhammad says Miller lies. Olga says Matteo lies. Merlin says Mallory lies. Teressa says Phoebe, Orin and Mario all tell the truth. Delbert says exactly two of Ryan, Oona and Mateo tells the truth. Melody says Marty lies. Fletcher says either exactly two of Kandi, Maverick and Ovid tell the truth or none of them. Mateo says Oona lies. Jaymie says either all three of Jamey, Osmond and Micah lie, or two of them tell the truth. Andree says Conception lies. Sherrie says exactly two of Elanor, Michelle and Orlando tells the truth. Octavio says Morgan lies. Ollie says Octavian tells the truth. Oswald says Murphy tells the truth. Micah says Osmond tells the truth. Helene says only one of Andree, Owen and Merlin lies. Ryan says exactly one of Inga, Miller and Muhammad tell the truth. Conception says Delbert lies. Ozzy says Mindy tells the truth. Orly says Murphy lies. Ovid says Maverick tells the truth. Jim says exactly two of Alexis, Mary and Mackenzie tell the truth. Ka says exactly two of Fidel, Matteo and Olga tells the truth. Martin says Maya lies. Tamika says only one of Amberly, Mustafa and Martin tells the truth. Olympia says Orson tells the truth. Osvaldo says Orson lies. Willian says Gwenn, Osborne and Molly all tell the truth. Owyn says Oden tells the truth. Fidel says Raymond, Oswald and Orly all tell the truth. Lalit says Morgan tells the truth. Mario says Melvin lies. Mustafa says Maya tells the truth. Owen says Mallory tells the truth. Raymond says Fletcher lies. Mackenzie says Mary lies. Oswin says Orla lies. Shalonda says either exactly two of Jim, Octavian and Ollie tell the truth or none of them. Osmond says Micah tells the truth. Oden says Owyn tells the truth. Orlando says Michelle lies. Michelle says Orlando lies. Gwenn says either all three of Helene, Owyn and Oden lie, or two of them tell the truth. Elanor says Shalonda tells the truth. Alexis says only one of Ka, Olympia and Osvaldo tells the truth. Mary says Mackenzie lies. Madison says Osian lies. Orin says Melvin tells the truth. Octavian says Ollie tells the truth. Alejandro says exactly two of Sherrie, Marty and Melody tells the truth. Matteo says Olga lies. Jamey lies. Marcus says Mindy lies. Amberly says only one of Alejandro, Mina and Oswin lies. Molly says Uri lies. Mina says Orla tells the truth. Osian says Madison lies. Oona says Mateo lies. Marty says Melody lies. Kandi says only one of Willian, Lalit and Octavio tells the truth. Phoebe says exactly two of Tamika, Osian and Madison tell the truth. Osborne says Uri tells the truth. Miller says Muhammad lies. Maverick says Ovid tells the truth. Do Sherrie, Amberly, and Teressa tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}