{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bf74000c-46ed-5c19-8e00-3b7db3d39ea4","task_key":"bbeh~5fweb~5fof~5flies--full--bf74000c-46ed-5c19-8e00-3b7db3d39ea4","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Oswin says Mason lies. Natasha says Ozias tells the truth. Obadiah says Millie lies. Jaymie says either all three of Rashida, Morgan and Ocean tell the truth or only one of them. Kandi says either exactly two of Fletcher, Merlin and Melvin tell the truth or none of them. Uri says Oona lies. Mackenzie says Matteo tells the truth. Osian says Onyx lies. Vina says only one of Amberly, Mavis and Osian tells the truth. Mark says Misty tells the truth. Ocean says Morgan tells the truth. Crista says either all three of Jim, Mackenzie and Matteo tell the truth or only one of them. Jerry says exactly one of Crista, Oswin and Mason tell the truth. Osric says Odin tells the truth. Mavis says Onyx tells the truth. Oona says Uri lies. Oda says Osborne lies. Odile says Millie tells the truth. Matteo says Mackenzie tells the truth. Morris says Mary tells the truth. Rashida says exactly two of Ka, Oona and Uri tell the truth. Fidel says Phoebe, Odile and Obadiah all tell the truth. Fletcher says either all three of Andree, Mary and Morris lie, or two of them tell the truth. Marshall says Misty lies. Melvin says Merlin tells the truth. Merlin says Melvin tells the truth. Morgan says Ocean tells the truth. Jim tells the truth. Mason says Oswin lies. Ka says Vina tells the truth. Osborne says Oda lies. Alejandro says either all three of Kandi, Odin and Osric lie, or two of them tell the truth. Phoebe says either all three of Christie, Natasha and Ozias lie, or two of them tell the truth. Ozias says Natasha tells the truth. Christie says Jerry tells the truth. Odin says Osric tells the truth. Andree says exactly two of Fidel, Osborne and Oda tells the truth. Amberly says Alejandro, Mark and Marshall all tell the truth. Mary says Morris tells the truth. Do Ka, Natasha, and Jaymie tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}