{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"609655f0-2b72-57e4-b5bd-828cc33aaef8","task_key":"bbeh~5fweb~5fof~5flies--full--609655f0-2b72-57e4-b5bd-828cc33aaef8","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Audrie says exactly two of Bernita, Ophelia and Murray tell the truth. Jamey says either all three of Inga, Oda and Milo lie, or two of them tell the truth. John says Molly lies. Mackenzie says Bahar tells the truth. Milo says Oda tells the truth. Nishanth says Muhammad lies. Ophelia says Murray lies. Oda says Milo tells the truth. Mavis says Morgan lies. Helene says either exactly one of Tamika, Mackenzie and Bahar tells the truth, or all three of them. Kristian says Sima, Oakley and Millie all tell the truth. Oakley says Mateo tells the truth. Sherrie says Rashida tells the truth. Sima says Shalonda lies. Inga says Teressa lies. Delfina says Helene tells the truth. Teressa says Kristian lies. Millie says Mateo lies. Natasha says Molly tells the truth. Disha says Mallory lies. Tamika says Audrie lies. Morgan says Mavis lies. Oran says Oona tells the truth. Vina says either all three of Delfina, Oona and Oran lie, or two of them tell the truth. Rashida says exactly one of Jamey, Nishanth and Muhammad tell the truth. Crista says only one of Alexis, Marina and Disha tells the truth. Bahar says Mackenzie tells the truth. Alexis says either exactly one of Sherrie, Michaela and Mariana tells the truth, or all three of them. Sal says Crista lies. Gwenn lies. Marina says Mallory tells the truth. Bernita says exactly one of Gwenn, Morgan and Mavis tells the truth.Mariana says Michaela tells the truth. Oona says Oran tells the truth. Michaela says Mariana tells the truth. Murray says Ophelia lies. Muhammad says Nishanth lies. Shalonda says Vina, Natasha and John all tell the truth. Do Teressa, Disha, and Sal tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}