{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b4f89bca-7d33-5a99-b485-8de3c874dd62","task_key":"bbeh~5fweb~5fof~5flies--full--b4f89bca-7d33-5a99-b485-8de3c874dd62","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Megan says Marina tells the truth. Phoebe lies. Odile says Mike tells the truth. Sima says only one of Phoebe, Orlando and Sanket lies. Hritik says Maximilian tells the truth. Malcolm says Mavis tells the truth. Shaunda says exactly two of Shenna, Marvin and Oswald tells the truth. Odette says Oran tells the truth. Jamey says Ka lies. Gwenn says either exactly one of Leda, Molly and Disha tells the truth, or all three of them. Mark says Lalit tells the truth. Kate says Otto tells the truth. Monica says Miriam lies. Andree says Ryan lies. Sanket says Ozzy lies. Alejandro says either exactly one of Audrie, Lalit and Mark tells the truth, or all three of them. Teressa says exactly two of Vernell, Miles and Orla tells the truth. Marina says Megan tells the truth. Oona says Oda lies. Orlando says Ozzy tells the truth. Molly says Disha tells the truth. Lalit says Mark tells the truth. Vernell says Lorine, Hritik and Octavian all tell the truth. Mallory says Meadow tells the truth. Audrie says either exactly one of Andree, Mavis and Malcolm tells the truth, or all three of them. Bernita says exactly two of Elanor, Oona and Oda tells the truth. Elanor says either exactly two of Fletcher, Meadow and Mallory tell the truth or none of them. Helene says either all three of Bernita, Oran and Odette lie, or two of them tell the truth. Mavis says Malcolm tells the truth. Ryan says only one of Teressa, Oakleigh and Monica tells the truth. Shenna says only one of Gwenn, Muhammad and Orson lies. Shalonda says Shaunda lies. Marvin says Oswald lies. Oswald says Marvin lies. Oleg says Mike lies. Orson says Natasha lies. Disha says Molly tells the truth. Miles says Orla lies. Oran says Odette tells the truth. Oda says Oona lies. Fletcher says either all three of Shalonda, Otto and Kate tell the truth or only one of them. Oakleigh says Miriam tells the truth. Otto says Kate tells the truth. Leda says either exactly two of Jamey, Megan and Marina tell the truth or none of them. Meadow says Mallory tells the truth. Crista says Jerry tells the truth. Octavian says Maximilian lies. Jerry says only one of Alejandro, Odile and Oleg tells the truth. Muhammad says Natasha tells the truth. Lorine says Sima lies. Ka says Crista lies. Orla says Miles lies. Do Leda, Oran, and Helene tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}