{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7fb57751-47f6-53c3-9552-8453db3b4944","task_key":"bbeh~5fweb~5fof~5flies--full--7fb57751-47f6-53c3-9552-8453db3b4944","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Ocean says Octavius tells the truth. Inga says exactly one of Ka, Odysseus and Orlando tell the truth. Mitchell says Mustafa lies. Melody says Martha tells the truth. Maurice says Odessa lies. Octavius says Ocean tells the truth. Orion says Oda lies. Oda says Orion lies. Marty says Odile tells the truth. Jamey says Shaunda, Olympia and Osmond all tell the truth. Virginia says Mavis lies. Oliver says Muhammad tells the truth. Opal says Octavia lies. Marina says Ovid lies. Muhammad says Oliver tells the truth. Christie says Dallas, Uri and Virginia all tell the truth. Phoebe says either all three of Conception, Marty and Odile tell the truth or only one of them. Orhan says Morgan tells the truth. Mustafa says Mitchell lies. Delbert says either all three of Fidel, Orhan and Morgan tell the truth or only one of them. McKenzie says Oleg tells the truth. Bahar says Oleg lies. Crista says only one of Vina, Oona and Maurice lies. Delfina says Rashida lies. Vina says Teressa lies. Tamika lies. Onyx says Mandy lies. Hritik says Ofelia lies. Fidel says exactly one of Sima, Ofelia and Hritik tells the truth.Mackenzie says Molly tells the truth. Rashida says Sal tells the truth. Marvin says Ovid tells the truth. Kandi says exactly one of Sherrie, Opal and Octavia tell the truth. Mickey says Milo tells the truth. Max says Maxim lies. Conception says Jamey lies. Bernita says Delbert lies. Shaunda says Yoland tells the truth. Osian says Milo lies. Odysseus says Orlando lies. Dallas says only one of Shalonda, Marvin and Marina tells the truth. Willian says either exactly one of Kandi, Muhammad and Oliver tells the truth, or all three of them. Martha says Melody tells the truth. Orlando says Odysseus lies. Antwan says Christie tells the truth. Ka says exactly one of Alexis, Mandy and Onyx tells the truth.Oona says Odessa tells the truth. Yoland says only one of Bernita, McKenzie and Bahar tells the truth. Alejandro says exactly two of Kristian, Orion and Oda tells the truth. Morgan says Orhan tells the truth. Osmond says Mario lies. Maxim says Max lies. Mandy says Onyx lies. Ofelia says Hritik lies. Shalonda says exactly two of Crista, Mitchell and Mustafa tells the truth. Michael says Molly lies. Sal says exactly one of Willian, Maxim and Max tells the truth.Uri says Mavis tells the truth. Sherrie says either all three of Alejandro, Martha and Melody lie, or two of them tell the truth. Teressa says either all three of Tamika, Octavius and Ocean lie, or two of them tell the truth. Kristian says Antwan tells the truth. Sima says only one of Delfina, Mackenzie and Michael lies. Olympia says Mario tells the truth. Octavia says Opal lies. Odile says Marty tells the truth. Alexis says Phoebe, Mickey and Osian all tell the truth. Do Willian, Delfina, and Inga tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}