{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6d2dd9c6-336e-5fad-9c4a-97bd2df7a9e8","task_key":"bbeh~5fweb~5fof~5flies--full--6d2dd9c6-336e-5fad-9c4a-97bd2df7a9e8","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Osborn says Odin lies. Miguel says Mason tells the truth. Othello says Mitchell tells the truth. Jamey says either exactly one of Willian, Megan and Mona tells the truth, or all three of them. Sima tells the truth. Willian says Sima, Othello and Bahar all tell the truth. Megan says Mona tells the truth. Yoland says either exactly one of Jaymie, Oona and Margaret tells the truth, or all three of them. Kristian says either all three of Jamey, Marsha and Michael lie, or two of them tell the truth. Maria says Ottoline lies. Teressa says only one of Jerry, Michelle and Octavio tells the truth. Ozias says Nicole tells the truth. Jim says either exactly one of Shenna, Milton and Marty tells the truth, or all three of them. Michelle says Maybelle tells the truth. Otto says Melody lies. Ovid says Marvin tells the truth. Delfina says either exactly two of Ryan, Ovid and Marvin tell the truth or none of them. Elanor says exactly two of Helene, Octavius and Odessa tells the truth. Marvin says Ovid tells the truth. Kandi says Raymond lies. Margaret says Oona tells the truth. Milton says Marty tells the truth. Odessa says Octavius lies. Maya says Olivia lies. Nicole says Ozias tells the truth. Bahar says Mitchell lies. Mona says Megan tells the truth. Meadow says Oakley tells the truth. Mason says Miguel tells the truth. Phoebe says Delfina lies. Merlin says Melinda tells the truth. Andree says exactly two of Teressa, Olivia and Maya tell the truth. Olivia says Maya lies. Melody says Otto lies. Ka says either exactly two of Andree, Mason and Miguel tell the truth or none of them. Raymond says exactly one of Elanor, Odin and Osborn tells the truth.Michael says Marsha tells the truth. Octavio says Maybelle lies. Oona says Margaret tells the truth. Molly says Oakley lies. Marty says Milton tells the truth. Shenna says only one of Yoland, Meredith and Maria lies. Helene says Phoebe, Merlin and Misty all tell the truth. Audrie says only one of Antwan, Meadow and Molly lies. Octavius says Odessa lies. Jerry says either exactly one of Jim, Ozias and Nicole tells the truth, or all three of them. Meredith says Ottoline tells the truth. Antwan says Kandi lies. Ryan says exactly two of Kristian, Otto and Melody tell the truth. Muriel says Odysseus lies. Odysseus says Muriel lies. Odin says Osborn lies. Marsha says Michael tells the truth. Misty says Melinda lies. Jaymie says exactly one of Audrie, Odysseus and Muriel tell the truth. Do Kandi, Ozias, and Ka tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}