{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7eb1df23-15ae-5402-92de-79b7f7cf91cd","task_key":"bbeh~5fweb~5fof~5flies--full--7eb1df23-15ae-5402-92de-79b7f7cf91cd","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Antwan says exactly one of Alexis, Kate and Matthew tells the truth.Maria says Mackenzie tells the truth. Marlon says Onyx tells the truth. Othello says Onyx lies. Kate says Matthew lies. Michael says Ocean lies. Orlando says Orion tells the truth. Tamika says Leda tells the truth. Orion says Orlando tells the truth. Olympia says Oakley lies. Olga says Martha tells the truth. Leda says exactly one of Jamey, Malcolm and Melanie tell the truth. Megan says Marsha lies. Shalonda says exactly two of Sima, Muhammad and Ophelia tells the truth. Matteo says Mallory lies. Osbert says Oden tells the truth. Melody says Mark lies. Mario says Omar tells the truth. Miller says Oakley tells the truth. Melanie says Malcolm lies. Shenna says Antwan, Moses and Maximilian all tell the truth. Dallas tells the truth. Malcolm says Melanie lies. Matthew says Kate lies. Marvin says Osmond tells the truth. Sherrie says either exactly one of Raymond, Marvin and Osmond tells the truth, or all three of them. Meredith says Mallory tells the truth. Martha says Olga tells the truth. Gwenn says exactly one of Sherrie, Oswin and Millicent tells the truth.Moses says Maxim tells the truth. Rashida says exactly two of Alejandro, Melody and Mark tells the truth. Raymond says only one of Tamika, Marlon and Othello lies. Jim says either exactly one of Rashida, Mackenzie and Maria tells the truth, or all three of them. Ophelia says Muhammad lies. Jaymie says exactly one of Jerry, Michael and Ocean tell the truth. Oswin says Millicent lies. Ka says Elanor lies. Owyn says Oda lies. Crista says exactly one of Shalonda, Oona and Miriam tells the truth.Marsha says Megan lies. Muhammad says Ophelia lies. Maximilian says Maxim lies. Omar says Mario tells the truth. Alejandro says exactly one of Vina, Megan and Marsha tell the truth. Ocean says Michael lies. Vina says either all three of Gwenn, Mario and Omar tell the truth or only one of them. Jamey says either all three of Audrie, Osbert and Oden lie, or two of them tell the truth. Willian says Jim lies. Sima says either exactly one of Ka, Martha and Olga tells the truth, or all three of them. Elanor says either all three of Kandi, Orlando and Orion tell the truth or only one of them. Alexis says Phoebe tells the truth. Mackenzie says Maria tells the truth. Miriam says Oona lies. Audrie says Shenna tells the truth. Olaf says Oda tells the truth. Millicent says Oswin lies. Mark says Melody lies. Oona says Miriam lies. Phoebe says Dallas, Olaf and Owyn all tell the truth. Oden says Osbert tells the truth. Jerry says only one of Crista, Meredith and Matteo lies. Kandi says only one of Willian, Miller and Olympia lies. Osmond says Marvin tells the truth. Do Elanor, Ka, and Jaymie tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}