{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"74b61d43-7242-5cfc-9c47-0e0da4c50d5d","task_key":"bbeh~5fweb~5fof~5flies--full--74b61d43-7242-5cfc-9c47-0e0da4c50d5d","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Jaymie says exactly two of Sima, Oakley and Maya tells the truth. Orion says Ollie lies. Sherrie says either all three of Delbert, Orin and Mason lie, or two of them tell the truth. Osborne says Ozias tells the truth. Milton says Olga lies. Lalit says Mallory lies. Millie says Maria tells the truth. Dallas says either all three of Elanor, Maria and Millie lie, or two of them tell the truth. Orin says Mason tells the truth. Andree says either exactly two of Rashida, Muriel and Orson tell the truth or none of them. Morris says Melinda tells the truth. Melinda says Morris tells the truth. Micah says Natasha lies. Otis says Miriam tells the truth. Teressa says exactly one of Fletcher, Milton and Olga tells the truth.Elanor says Kristian, Onyx and Marcus all tell the truth. Christie says Raymond lies. Lorine says Tamika tells the truth. Uri says Mitchell lies. Odette says Sanket tells the truth. Orly says Octavia lies. Marina says Oprah lies. Conception says either exactly one of Ka, Miriam and Otis tells the truth, or all three of them. Maya says Oakley lies. Sima says exactly one of Christie, Mallory and Lalit tell the truth. Raymond says either exactly two of Dallas, Mike and Miranda tell the truth or none of them. Helene says either exactly two of Crista, Oleg and Megan tell the truth or none of them. Mike says Miranda tells the truth. Mason says Orin tells the truth. Oleg says Megan tells the truth. Kristian says exactly two of Willian, Micah and Natasha tell the truth. Leda says exactly two of Kandi, Marina and Oprah tells the truth. Rashida says only one of Bernita, Mickey and Orion tells the truth. Marcus says Owyn lies. Octavio says Mitchell tells the truth. Antwan says either exactly one of Conception, Morris and Melinda tells the truth, or all three of them. Muriel says Orson tells the truth. Mercedes says Murray lies. Bernita says Alejandro tells the truth. Ka says only one of Jaymie, Miguel and Osbert lies. Kandi says Sherrie lies. Maynard says Mariana tells the truth. Mickey says Ollie tells the truth. Crista says either all three of Inga, Odette and Sanket lie, or two of them tell the truth. Jamey says either exactly two of Sal, Meredith and Maxim tell the truth or none of them. Maxim says Meredith tells the truth. Oprah says Marina lies. Olga says Milton lies. Sal says Andree lies. Miranda says Mike tells the truth. Osbert says Marlon lies. Mallory says Lalit lies. Maria says Millie tells the truth. Orson says Muriel tells the truth. Murray says Mercedes lies. Phoebe says either all three of Jamey, Osborne and Ozias lie, or two of them tell the truth. Sanket says Odette tells the truth. Meredith says Maxim tells the truth. Onyx says Owyn tells the truth. Tamika says Jim lies. Delbert says exactly one of Phoebe, Octavia and Orly tell the truth. Inga says exactly one of Teressa, Mercedes and Murray tell the truth. Megan says Oleg tells the truth. Ozias says Osborne tells the truth. Fletcher says either exactly one of Leda, Maynard and Mariana tells the truth, or all three of them. Miguel says Marlon tells the truth. Oakley says Maya lies. Mariana says Maynard tells the truth. Jim lies. Natasha says Micah lies. Miriam says Otis tells the truth. Octavia says Orly lies. Willian says Lorine tells the truth. Alejandro says only one of Antwan, Octavio and Uri lies. Do Teressa, Kandi, and Helene tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}