{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1ab6b673-d2a8-55ab-83f7-6fd8f5d1077a","task_key":"bbeh~5fweb~5fof~5flies--full--1ab6b673-d2a8-55ab-83f7-6fd8f5d1077a","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Sima says exactly one of Fidel, Murphy and Otis tells the truth.Molly says Milo tells the truth. Murphy says Otis lies. Madison says Oriana tells the truth. Jim says exactly one of Inga, Olympia and Miriam tells the truth.Oona says Micah lies. Octavia says Mark lies. Otis says Murphy lies. Jaymie says either all three of Jamey, Madison and Oriana tell the truth or only one of them. Phoebe says only one of Alejandro, Bahar and Osborn tells the truth. Marsha says Muhammad lies. Teressa says only one of Sal, Maurice and Oscar lies. Audrie tells the truth. Olympia says Miriam lies. Oakleigh says Mallory lies. Alejandro says Delbert, John and Octavia all tell the truth. Inga says Kandi tells the truth. John says Mark tells the truth. Margaret says Kate lies. Rashida says Teressa lies. Matteo says Marvin lies. Marlon says Oswald lies. Lorine says Rashida lies. Jamey says only one of Tamika, Maya and Melvin tells the truth. Shaunda says only one of Jim, Odin and Matteo lies. Octavian says Kate tells the truth. Ofelia says Micah tells the truth. Orin says Misty lies. Maximilian says Oakley lies. Maya says Ottoline tells the truth. Yoland says exactly two of Ka, Oakley and Maximilian tells the truth. Opal says Osian tells the truth. Osian says Opal tells the truth. Elanor says exactly two of Conception, Oakleigh and Mallory tell the truth. Misty says Orin lies. Crista says either exactly one of Yoland, Natasha and Mina tells the truth, or all three of them. Natasha says Mina tells the truth. Conception says Audrie tells the truth. Maurice says Maxim tells the truth. Oswin says Muhammad tells the truth. Sal says only one of Elanor, Ofelia and Oona lies. Delbert says only one of Jerry, Oswin and Marsha lies. Ka says Phoebe lies. Alexis says either all three of Jaymie, Osian and Opal tell the truth or only one of them. Fidel says exactly two of Crista, Misty and Orin tells the truth. Oakley says Maximilian lies. Bahar says Maverick tells the truth. Mallory says Oakleigh lies. Delfina says Lorine, Octavian and Margaret all tell the truth. Melvin says Ottoline lies. Kandi says either exactly two of Alexis, Milo and Molly tell the truth or none of them. Tamika says Delfina tells the truth. Osborn says Maverick lies. Miriam says Olympia lies. Jerry says exactly two of Shaunda, Oswald and Marlon tell the truth. Milo says Molly tells the truth. Oscar says Maxim lies. Oswald says Marlon lies. Odin says Marvin tells the truth. Mina says Natasha tells the truth. Oriana says Madison tells the truth. Do Jaymie, Alexis, and Sima tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}