{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9b0fb398-d146-5056-b7ff-2fd0db9fbce2","task_key":"bbeh~5fweb~5fof~5flies--full--9b0fb398-d146-5056-b7ff-2fd0db9fbce2","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Antwan says exactly one of Delfina, Moses and Opal tells the truth.Odile says Olga lies. Teressa says either exactly two of Leda, Osian and Octavia tell the truth or none of them. Octavia says Osian tells the truth. Olive says Olympia lies. Nishanth says Owyn tells the truth. Jaymie says either exactly two of Kristian, Nishanth and Owyn tell the truth or none of them. Osvaldo says Osborne lies. Kristian says Gwenn lies. Ophelia says Oswin tells the truth. Opal says Moses lies. Jamey says exactly one of Alexis, Maximilian and Mary tells the truth.Owyn says Nishanth tells the truth. Delfina says either exactly two of Bernita, Miller and Lalit tell the truth or none of them. Alexis says only one of Delbert, Ophelia and Olaf lies. Osian says Octavia tells the truth. Lalit says Miller tells the truth. Bahar says Osborne tells the truth. Moses says Opal lies. Dallas says only one of Teressa, Mia and Ocean tells the truth. Miller says Lalit tells the truth. Mary says Maximilian lies. Orson says Maverick tells the truth. Yoland says Inga tells the truth. Andree says exactly two of Antwan, Olympia and Olive tell the truth. Orion says Otis lies. Ocean says Octavius lies. Mina says Marsha lies. Gwenn says either exactly two of Fidel, Martha and Ollie tell the truth or none of them. Miles says Marsha tells the truth. Olga says Odile lies. Bernita says Sherrie lies. Ozias says Oona tells the truth. Olaf says Oswin lies. Sherrie says Jaymie tells the truth. Leda says Andree tells the truth. Alejandro says exactly one of Shalonda, Otis and Orion tell the truth. Marty says Morris tells the truth. Vernell says Dallas, Miles and Mina all tell the truth. Maximilian says Mary lies. Vina says exactly one of Jerry, Mason and Melvin tell the truth. Shenna says Kandi lies. Otis says Orion lies. Sal says Ryan lies. Fidel says only one of Yoland, Orson and Marcus lies. Marcus says Maverick lies. Jerry says only one of Jamey, Bahar and Osvaldo lies. Inga tells the truth. Mason says Melvin lies. Olympia says Olive lies. Shalonda says Sima lies. Ollie says Martha tells the truth. Sima says Vernell tells the truth. Kandi says exactly two of Vina, Odile and Olga tells the truth. Ryan says either exactly one of Alejandro, Morris and Marty tells the truth, or all three of them. Minerva says Oona lies. Delbert says Jim lies. Martha says Ollie tells the truth. Melvin says Mason lies. Mia says Octavius tells the truth. Jim says only one of Sal, Ozias and Minerva lies. Morris says Marty tells the truth. Do Jim, Miles, and Shenna tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}