{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"52b46d87-14ad-58bf-914a-515fa9faa020","task_key":"bbeh~5fweb~5fof~5flies--full--52b46d87-14ad-58bf-914a-515fa9faa020","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Maurice says Millicent lies. Phoebe says Fletcher tells the truth. John says Virginia lies. Melody says Morgan tells the truth. Meadow says Mario tells the truth. Fletcher says either all three of Andree, Maxim and Obadiah tell the truth or only one of them. Octavius says Opal lies. Melissa says Mickey lies. Marvin says Molly lies. Mehran says Milton tells the truth. Kandi says only one of Willian, Meadow and Orlando tells the truth. Vina says either all three of Delbert, Minnie and Mona tell the truth or only one of them. Ofelia says Milton lies. Otto says Marina lies. Morgan says Melody tells the truth. Jaymie says exactly one of Gwenn, Melissa and Mickey tell the truth. Orlando says Mario lies. Molly says Marvin lies. Virginia says John lies. Jerry says exactly two of Shenna, Marina and Otto tells the truth. Oona says Miranda tells the truth. Sima says Leda, Kate and Martha all tell the truth. Obadiah says Maxim tells the truth. Ottoline says Odile lies. Ka says only one of Raymond, Michaela and Madison tells the truth. Alexis says exactly two of Sima, Virginia and John tells the truth. Bernita says Jerry lies. Leda says exactly two of Kandi, Odile and Ottoline tell the truth. Tamika says exactly two of Ka, Ozzie and Maybelle tell the truth. Madison says Osvaldo lies. Omar says Miranda lies. Odell says Ophelia tells the truth. Sal says either all three of Lorine, Nishanth and Oakley tell the truth or only one of them. Ozzie says Maybelle lies. Willian lies. Gwenn says either all three of Sal, Morgan and Melody tell the truth or only one of them. Oakley says Nishanth tells the truth. Marina says Otto lies. Ophelia says Odell tells the truth. Mariah says Millicent tells the truth. Kate says Morris tells the truth. Elanor says either exactly one of Alexis, Odell and Ophelia tells the truth, or all three of them. Orion says Opal tells the truth. Andree says only one of Vina, Mariah and Maurice lies. Raymond says only one of Jaymie, Mehran and Ofelia lies. Odile says Ottoline lies. Lorine says only one of Elanor, Orion and Octavius tells the truth. Mona says Minnie tells the truth. Minnie says Mona tells the truth. Maybelle says Ozzie lies. Octavia says Oakleigh lies. Shenna says exactly two of Ryan, Oakleigh and Octavia tells the truth. Maxim says Obadiah tells the truth. Michaela says Osvaldo tells the truth. Ryan says exactly two of Vernell, Marvin and Molly tells the truth. Mickey says Melissa lies. Delbert says only one of Tamika, Oona and Omar lies. Vernell says Phoebe lies. Nishanth says Oakley tells the truth. Martha says Morris lies. Oakleigh says Octavia lies. Do Vernell, Jerry, and Bernita tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}