{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b0c839b3-3eea-591e-86df-01a3081bedce","task_key":"bbeh~5fweb~5fof~5flies--full--b0c839b3-3eea-591e-86df-01a3081bedce","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Rashida says exactly one of Helene, Hritik and Nicole tell the truth. Melissa says John lies. Nicole says Hritik lies. Marvin says Orhan tells the truth. Sanket says Disha tells the truth. Shalonda says either exactly one of Sima, Mark and Misty tells the truth, or all three of them. Ka says either all three of Gwenn, Marvin and Orhan tell the truth or only one of them. Mark says Misty tells the truth. Moses says Orlando tells the truth. Disha says Sanket tells the truth. Mandy says Orly tells the truth. Misty says Mark tells the truth. Antwan says either all three of Dallas, Disha and Sanket lie, or two of them tell the truth. Andree says either exactly two of Shenna, Morgan and Olive tell the truth or none of them. Maybelle says Ofelia lies. Alexis says Teressa tells the truth. Conception says either all three of Ka, Orly and Mandy tell the truth or only one of them. Olympia says Matteo tells the truth. Maverick says Mike lies. Fidel says either exactly two of Leda, Olympia and Matteo tell the truth or none of them. Helene says exactly one of Willian, Ozzie and Owyn tells the truth.Martin says Mateo tells the truth. Jamey says either all three of Yoland, Odette and Otis lie, or two of them tell the truth. Matteo says Olympia tells the truth. Orly says Mandy tells the truth. Elanor says only one of Antwan, Octavius and Octavio lies. Odette says Otis tells the truth. Otis says Odette tells the truth. Olive says Morgan tells the truth. Dallas says exactly two of Rashida, John and Melissa tells the truth. Leda says exactly two of Alexis, Maybelle and Ofelia tell the truth. Shaunda says Andree tells the truth. Omar says Marty tells the truth. Octavius says Mackenzie tells the truth. Sima says exactly one of Conception, Michaela and Bahar tell the truth. Hritik says Nicole lies. John says Melissa lies. Mateo says Martin tells the truth. Mike says Maverick lies. Yoland says either exactly one of Shalonda, Mateo and Martin tells the truth, or all three of them. Michaela says Bahar lies. Teressa says only one of Shaunda, Moses and Uri tells the truth. Bernita says exactly one of Jamey, Maverick and Mike tell the truth. Owyn says Ozzie lies. Bahar says Michaela lies. Uri says Orlando lies. Ozzie says Owyn lies. Gwenn says either all three of Fidel, Marty and Omar lie, or two of them tell the truth. Octavio says Mackenzie lies. Shenna says Elanor tells the truth. Orhan says Marvin tells the truth. Morgan says Olive tells the truth. Ofelia says Maybelle lies. Willian lies. Marty says Omar tells the truth. Do Conception, Leda, and Bernita tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}