{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c10c91c4-9828-5c44-b2f4-0f1257bd4d69","task_key":"bbeh~5fweb~5fof~5flies--full--c10c91c4-9828-5c44-b2f4-0f1257bd4d69","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Shaunda says exactly two of Shenna, Olympia and Olaf tell the truth. Jim lies. Malcolm says Merlin tells the truth. Mehran says Ophelia lies. Olympia says Olaf lies. Delbert says either exactly two of Shaunda, Orla and Olivia tell the truth or none of them. Moses says Nicole lies. Minnie says Uri tells the truth. Fidel says exactly one of Elanor, Mia and Ozzy tell the truth. Phoebe says only one of Willian, Owyn and Moses lies. Olivia says Orla tells the truth. Virginia says Mason lies. Mia says Ozzy lies. Audrie says either all three of Kristian, Megan and John tell the truth or only one of them. Oden says Marina lies. John says Megan tells the truth. Mario says Ophelia tells the truth. Osmond says Marina tells the truth. Murphy says Millie tells the truth. Marcus says Michael tells the truth. Christie says Fidel tells the truth. Martha says Ozzie tells the truth. Sima says only one of Phoebe, Minnie and Morgan lies. Jaymie says either all three of Antwan, Othello and Mona tell the truth or only one of them. Octavius says Orson lies. Morgan says Uri lies. Ozzie says Martha tells the truth. Oda says Marshall tells the truth. Maverick says Odessa lies. Dallas says either exactly one of Jim, Millie and Murphy tells the truth, or all three of them. Shenna says only one of Amberly, Mario and Mehran lies. Opal says Mark lies. Merlin says Malcolm tells the truth. Ryan says exactly one of Yoland, Miller and Oakleigh tells the truth.Antwan says either all three of Christie, Marcus and Michael lie, or two of them tell the truth. Marshall says Oda tells the truth. Willian says exactly one of Delbert, Marsha and Oriana tells the truth.Conception says Leda, Oswald and Maverick all tell the truth. Delfina says either exactly two of Jaymie, Orion and Obadiah tell the truth or none of them. Millie says Murphy tells the truth. Othello says Mona tells the truth. Oakleigh says Miller lies. Oswald says Odessa tells the truth. Michael says Marcus tells the truth. Odysseus says Melody tells the truth. Kristian says either all three of Sima, Malcolm and Merlin tell the truth or only one of them. Orson says Octavius lies. Melody says Odysseus tells the truth. Lorine says exactly two of Bernita, Orson and Octavius tell the truth. Yoland says only one of Tamika, Kate and Opal lies. Leda says exactly two of Lorine, Mason and Virginia tell the truth. Owyn says Nicole tells the truth. Oriana says Marsha lies. Elanor says either all three of Ryan, Martha and Ozzie tell the truth or only one of them. Obadiah says Orion tells the truth. Bernita says Audrie tells the truth. Kate says Mark tells the truth. Megan says John tells the truth. Amberly says only one of Dallas, Osmond and Oden tells the truth. Olaf says Olympia lies. Mona says Othello tells the truth. Jerry says either exactly one of Conception, Marshall and Oda tells the truth, or all three of them. Ozzy says Mia lies. Marsha says Oriana lies. Orla says Olivia tells the truth. Tamika says either exactly one of Jerry, Odysseus and Melody tells the truth, or all three of them. Miller says Oakleigh lies. Orion says Obadiah tells the truth. Mason says Virginia lies. Do Antwan, Marshall, and Delfina tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}