{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"db631efe-bfc7-5331-b3f2-3ae7b89a700f","task_key":"bbeh~5fweb~5fof~5flies--full--db631efe-bfc7-5331-b3f2-3ae7b89a700f","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Madison says Ophelia tells the truth. Mona says Melinda lies. Audrie says exactly one of Leda, Maybelle and Owyn tells the truth.Alexis says only one of Jamey, Kate and Max lies. Millicent says Melissa tells the truth. Muriel says Uri lies. Melissa says Millicent tells the truth. Mitchell says Miles tells the truth. Ophelia says Madison tells the truth. Elanor says either all three of Antwan, Ophelia and Madison tell the truth or only one of them. Mateo says Miranda tells the truth. Ryan says exactly one of Bernita, Osborne and Mike tell the truth. Minerva says Oprah lies. Obadiah says Murray lies. Octavia says Molly lies. Lorine says either exactly one of Ka, Madeleine and Misty tells the truth, or all three of them. Madeleine says Misty tells the truth. Kristian says either all three of Dallas, Odell and Otto lie, or two of them tell the truth. Misty says Madeleine tells the truth. Oriana says Malcolm tells the truth. Jamey says Helene tells the truth. Orhan says Miller lies. Teressa says either exactly one of Lorine, Olive and Odile tells the truth, or all three of them. Shenna says exactly one of Inga, Muriel and Uri tell the truth. Sal says either exactly one of Yoland, Oriana and Malcolm tells the truth, or all three of them. Otto says Odell tells the truth. Maybelle says Owyn lies. Inga says Audrie tells the truth. Willian says Kristian tells the truth. Ozias says Olga tells the truth. Leda says exactly two of Teressa, Murray and Obadiah tells the truth. Odile says Olive tells the truth. Kate says Morgan tells the truth. Orion says Melinda tells the truth. Dallas says exactly two of Phoebe, Molly and Octavia tells the truth. Oprah says Minerva lies. Bernita says exactly one of Sal, Oprah and Minerva tells the truth.Miles says Mitchell tells the truth. Miranda says Mateo tells the truth. Olive says Odile tells the truth. Antwan says Willian tells the truth. Otis says Olga lies. Ka says Elanor, Olympia and Orhan all tell the truth. Osborne says Mike lies. Mike says Osborne lies. Max says Morgan lies. Gwenn says either exactly one of Shenna, Millicent and Melissa tells the truth, or all three of them. Odell says Otto tells the truth. Helene says only one of Shalonda, Ozias and Otis lies. Uri says Muriel lies. Yoland lies. Owyn says Maybelle lies. Malcolm says Oriana tells the truth. Phoebe says Jerry, Orion and Mona all tell the truth. Jerry says either all three of Alexis, Mateo and Miranda tell the truth or only one of them. Molly says Octavia lies. Olympia says Miller tells the truth. Shalonda says either all three of Ryan, Miles and Mitchell lie, or two of them tell the truth. Murray says Obadiah lies. Do Lorine, Madison, and Gwenn tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}