{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"191ae777-07aa-5e01-97ca-69a3072cace3","task_key":"bbeh~5fweb~5fof~5flies--full--191ae777-07aa-5e01-97ca-69a3072cace3","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Ozzy says Octavius lies. Maurice says Oprah tells the truth. Octavio says Matthew tells the truth. Morris says Mina lies. Orion says Malcolm tells the truth. Orlando says Nicole tells the truth. Matthew says Octavio tells the truth. Mario says Miles tells the truth. Elanor says Bernita tells the truth. Marina says Maynard lies. Willian says exactly two of Jerry, Mina and Morris tells the truth. Amberly says exactly two of Gwenn, Oran and Millicent tell the truth. Onyx says Maya lies. Mitchell says Nishanth tells the truth. Miles says Mario tells the truth. Margaret says Minerva lies. Christie says exactly one of Fidel, Marina and Maynard tell the truth. Otis says Octavian tells the truth. Octavius says Ozzy lies. Olga says Oriana lies. Vina says Audrie tells the truth. Jerry says only one of Alexis, Maurice and Martin lies. Lorine says either exactly two of Conception, Octavian and Otis tell the truth or none of them. Conception says Ka tells the truth. Oran says Millicent lies. Bernita says either all three of Jaymie, Orion and Malcolm tell the truth or only one of them. Octavian says Otis tells the truth. Shalonda says exactly two of Lorine, Oriana and Olga tells the truth. Maya says Onyx lies. Teressa says exactly one of Andree, Ozzy and Octavius tells the truth.Michaela says Mustafa tells the truth. Sal says either exactly two of Rashida, Mario and Miles tell the truth or none of them. Mustafa says Michaela tells the truth. Molly says Nicole lies. Mina says Morris lies. Milton says Mercedes lies. Malcolm says Orion tells the truth. Alexis says either exactly two of Teressa, Michaela and Mustafa tell the truth or none of them. Ryan says exactly two of Shalonda, Onyx and Maya tells the truth. Andree says Jim, Orlando and Molly all tell the truth. Audrie says either exactly one of Ryan, Matthew and Octavio tells the truth, or all three of them. Shaunda says only one of Amberly, Melvin and Millie lies. Martin says Oprah lies. Jim says Vina lies. Raymond tells the truth. Ka says exactly one of Christie, Margaret and Minerva tells the truth.Jaymie says Raymond tells the truth. Mercedes says Milton lies. Oriana says Olga lies. Millie says Miller lies. Melvin says Miller tells the truth. Millicent says Oran lies. Nishanth says Mitchell tells the truth. Fidel says exactly two of Shaunda, Milton and Mercedes tell the truth. Maynard says Marina lies. Gwenn says Sal lies. Minerva says Margaret lies. Rashida says either all three of Elanor, Mitchell and Nishanth tell the truth or only one of them. Do Lorine, Christie, and Willian tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}