{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9b78a0b0-410f-53e9-a393-8ff2a993e7df","task_key":"bbeh~5fweb~5fof~5flies--full--9b78a0b0-410f-53e9-a393-8ff2a993e7df","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Gwenn says Delfina lies. Madison says Olivia tells the truth. Odessa says Virginia lies. Natasha says Osric tells the truth. Ophelia says Meadow tells the truth. Sherrie says either exactly one of Gwenn, Natasha and Osric tells the truth, or all three of them. Disha says Orlando lies. Mickey says Olivia lies. Amberly says either exactly one of Shaunda, Mohamed and Sanket tells the truth, or all three of them. Ozzy says Olympia lies. Mona says Molly tells the truth. Vernell says exactly two of Jaymie, Orla and Oakleigh tell the truth. Elanor says exactly two of Helene, Olympia and Ozzy tell the truth. Martha says Muhammad tells the truth. Oakleigh says Orla lies. Andree says only one of Bernita, Odette and Oden lies. Osric says Natasha tells the truth. Orla says Oakleigh lies. Virginia says Odessa lies. Inga says either all three of Elanor, Bahar and Oprah tell the truth or only one of them. Oden says Matthew lies. Mohamed says Sanket tells the truth. Otto says Orlando tells the truth. Shaunda says exactly one of Lorine, Odessa and Virginia tell the truth. Lorine says only one of Conception, Mona and Max lies. Maya says Owen lies. Jerry says only one of Christie, Madison and Mickey lies. Jaymie says Kandi tells the truth. Delfina says exactly one of Jerry, Maya and Owen tells the truth.Oprah says Bahar tells the truth. Bernita says only one of Amberly, Otto and Disha lies. Crista lies. Odette says Matthew tells the truth. Max says Molly lies. Conception says Inga tells the truth. Kandi says only one of Andree, Mariana and Monica tells the truth. Monica says Mehran lies. Mariana says Mehran tells the truth. Sanket says Mohamed tells the truth. Muhammad says Martha tells the truth. Ka says Sherrie tells the truth. Helene says only one of Ka, Ophelia and Maximilian tells the truth. Maximilian says Meadow lies. Raymond says exactly two of Crista, John and Oda tells the truth. Christie says either exactly one of Raymond, Martha and Muhammad tells the truth, or all three of them. Olympia says Ozzy lies. John says Oda lies. Oda says John lies. Owen says Maya lies. Bahar says Oprah tells the truth. Do Kandi, Disha, and Vernell tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}