{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"722acb98-1677-5a34-a7c4-4150bd1c3f8f","task_key":"bbeh~5fweb~5fof~5flies--full--722acb98-1677-5a34-a7c4-4150bd1c3f8f","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Omar says Octavio lies. Mateo says Sanket tells the truth. Octavio says Omar lies. Oriana says Oden tells the truth. Ottoline says Oliver lies. Fidel says exactly one of Jamey, Ottoline and Oliver tell the truth. Madison says Mina tells the truth. Otis says Miles tells the truth. Alexis says either all three of Kandi, Melvin and Mavis tell the truth or only one of them. Phoebe says Sima lies. Mia says Miguel lies. Sal says Alexis tells the truth. Olaf says Osmond lies. Sima says either all three of Sal, Millie and Onyx tell the truth or only one of them. Marty says McKenzie lies. Melvin says Mavis tells the truth. Marsha says Miriam lies. Sanket says Mateo tells the truth. Conception says exactly one of Delbert, Osric and Orly tells the truth.Teressa says Fidel lies. Oona says Odin lies. Oliver says Ottoline lies. Virginia says Miguel tells the truth. Lorine says either all three of Elanor, Miles and Otis tell the truth or only one of them. Jamey says Shaunda tells the truth. Margaret says Octavian tells the truth. Miriam says Marsha lies. Mark says Moses lies. Raymond says Ka lies. Olympia says Matthew tells the truth. Orly says Osric lies. Shaunda says exactly two of Conception, Miriam and Marsha tell the truth. Jerry says only one of Vernell, Olympia and Melissa tells the truth. Onyx says Millie tells the truth. Octavian says Margaret tells the truth. Ryan says either exactly one of Amberly, Oriana and Oden tells the truth, or all three of them. Shalonda says Kristian tells the truth. Kristian says Phoebe lies. Oakleigh says Disha lies. Sherrie tells the truth. Matteo says Odin tells the truth. Mina says Madison tells the truth. Osric says Orly lies. McKenzie says Marty lies. Millie says Onyx tells the truth. Rashida says only one of Shalonda, Virginia and Mia lies. Audrie says either exactly two of Jerry, Mateo and Sanket tell the truth or none of them. Mavis says Melvin tells the truth. Fletcher says exactly two of Rashida, McKenzie and Marty tell the truth. Kandi says exactly one of Gwenn, Octavio and Omar tells the truth.Gwenn says Raymond lies. Oden says Oriana tells the truth. Bernita says either exactly one of Audrie, Octavian and Margaret tells the truth, or all three of them. Amberly says either exactly one of Jaymie, Madison and Mina tells the truth, or all three of them. Elanor says only one of Fletcher, Orin and Olaf lies. Miles says Otis tells the truth. Orin says Osmond tells the truth. Disha says Oakleigh lies. Delfina says exactly two of Sherrie, Mark and Moses tell the truth. Vernell says Teressa lies. Melissa says Matthew lies. Ka says only one of Bernita, Matteo and Oona lies. Jaymie says exactly one of Delfina, Oakleigh and Disha tell the truth. Delbert says Ryan tells the truth. Moses says Mark lies. Do Kristian, Miles, and Lorine tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}