{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f1206173-0286-5e03-94aa-f2c166f80495","task_key":"bbeh~5fweb~5fof~5flies--full--f1206173-0286-5e03-94aa-f2c166f80495","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Minnie says Mackenzie tells the truth. Dallas says either exactly one of Phoebe, Mario and Marshall tells the truth, or all three of them. Conception says Sherrie tells the truth. Alexis says exactly two of Jim, Millicent and Oda tells the truth. Jerry says Shenna tells the truth. Mackenzie says Minnie tells the truth. Mina says Mona lies. Oda says Millicent lies. Odysseus says Oakleigh lies. Shalonda says Jerry tells the truth. Maria says Miller lies. Oprah says Uri lies. Melinda says Osbert lies. Mia says Oleg lies. Octavia says Muhammad tells the truth. Odette says Mateo tells the truth. Hritik says Oakleigh tells the truth. Mario says Marshall tells the truth. Ka says only one of Christie, Odessa and Owen lies. Mateo says Odette tells the truth. Vina says either exactly two of Conception, Ollie and Melody tell the truth or none of them. Alejandro says either all three of Rashida, Michael and Olive lie, or two of them tell the truth. Orion says Margaret lies. Michael says Olive tells the truth. Shenna says Yoland, Mary and Melinda all tell the truth. Christie says Delfina tells the truth. Raymond says either exactly one of Kristian, Oona and Natasha tells the truth, or all three of them. Jim says exactly two of Dallas, Oprah and Uri tells the truth. Nishanth says Micah tells the truth. Owen says Orly lies. Yoland says only one of Lorine, Mindy and Ozias tells the truth. Oona says Natasha tells the truth. Melody says Ollie tells the truth. Gwenn lies. Miller says Maria lies. Ozias says Osian lies. Marshall says Mario tells the truth. Oleg says Mia lies. Sherrie says only one of Ka, Hritik and Odysseus tells the truth. Ryan says only one of Shalonda, Octavia and Marvin lies. Natasha says Oona tells the truth. Sima says exactly one of Vina, Mina and Mona tells the truth.Phoebe says exactly two of Raymond, Orion and Margaret tells the truth. Kristian says either exactly two of Teressa, Minnie and Mackenzie tell the truth or none of them. Rashida says either exactly one of Ryan, Mateo and Odette tells the truth, or all three of them. Mona says Mina lies. Marvin says Muhammad lies. Teressa says exactly two of Alejandro, Orlando and Odin tell the truth. Orlando says Odin lies. Fletcher says either all three of Sima, Micah and Nishanth tell the truth or only one of them. Uri says Oprah lies. Mindy says Osian tells the truth. Odessa says Orly tells the truth. Millicent says Oda lies. Margaret says Orion lies. Lorine says exactly one of Fletcher, Maria and Miller tell the truth. Micah says Nishanth tells the truth. Odin says Orlando lies. Mary says Osbert tells the truth. Ollie says Melody tells the truth. Delfina says exactly two of Gwenn, Mia and Oleg tells the truth. Olive says Michael tells the truth. Do Shalonda, Mia, and Alexis tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}