{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6992361b-719b-5bae-9344-cbee7caf7e6a","task_key":"bbeh~5fweb~5fof~5flies--full--6992361b-719b-5bae-9344-cbee7caf7e6a","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Osian says Hritik lies. Orla says Kate lies. Alexis says exactly two of Jaymie, Mona and John tells the truth. Oswald says Uri tells the truth. Gwenn says Sal lies. Dallas says only one of Ryan, Mitchell and Oprah lies. Mandy says Oswin tells the truth. Ozias says Oakley lies. Olympia says Sanket tells the truth. Kandi says exactly one of Gwenn, Octavia and Ocean tells the truth.Sanket says Olympia tells the truth. Mitchell says Owen tells the truth. Mona says John lies. Ozzie says Opal tells the truth. Leda says exactly two of Jim, Ozias and Oakley tell the truth. Marcus says Mariana lies. Minerva says Micah lies. Willian says only one of Bernita, Mandy and Mason lies. Melvin says Hritik tells the truth. Sima says either exactly two of Teressa, Maynard and Osborn tell the truth or none of them. Virginia says Odell tells the truth. Oakley says Ozias lies. Jim says either all three of Lorine, Madison and Melanie lie, or two of them tell the truth. Lorine says either exactly one of Phoebe, Sanket and Olympia tells the truth, or all three of them. Antwan says exactly two of Jamey, Minerva and Micah tells the truth. Micah says Minerva lies. Odell says Virginia tells the truth. Ocean says Octavia lies. Melanie says Madison tells the truth. Fletcher says Andree tells the truth. Osborn says Maynard tells the truth. Shalonda says either exactly one of Fletcher, Opal and Ozzie tells the truth, or all three of them. Kate says Orla lies. Opal says Ozzie tells the truth. Vernell says Leda tells the truth. Sal tells the truth. Maynard says Osborn tells the truth. Jamey says either exactly two of Shenna, Oswald and Uri tell the truth or none of them. Teressa says Alexis lies. Octavia says Ocean lies. Oprah says Owen lies. Mariana says Marcus lies. Crista says Willian tells the truth. Meredith says Michaela lies. Mavis says Michaela tells the truth. Jaymie says exactly two of Antwan, Kate and Orla tells the truth. Madison says Melanie tells the truth. Ryan says either all three of Sima, Virginia and Odell lie, or two of them tell the truth. Mason says Oswin lies. Andree says Vernell, Melvin and Osian all tell the truth. Shenna says exactly two of Kandi, Marcus and Mariana tell the truth. Bernita says Dallas lies. John says Mona lies. Phoebe says only one of Crista, Mavis and Meredith lies. Uri says Oswald tells the truth. Do Willian, Marcus, and Shalonda tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}