{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"87018315-bbda-5a66-81d8-fbe6853ea55f","task_key":"bbeh~5fweb~5fof~5flies--full--87018315-bbda-5a66-81d8-fbe6853ea55f","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Mariana says Othello lies. Olaf says Oakleigh tells the truth. Vernell says only one of Fidel, Orion and Osbert lies. Osbert says Morgan lies. Lorine says Vernell tells the truth. Ollie says Orhan tells the truth. Octavia says Mark lies. Orla says Orlando lies. Phoebe says either exactly two of Fletcher, Mona and Mackenzie tell the truth or none of them. Willian says exactly one of Audrie, Mark and Octavia tells the truth.Miriam says Odile lies. Fidel says Willian, Orin and Maria all tell the truth. Fletcher tells the truth. Mona says Mackenzie tells the truth. Othello says Mariana lies. Murray says Muhammad lies. Orin says Octavius tells the truth. Marina says Melinda tells the truth. Orly says McKenzie lies. Kristian says exactly one of Ka, Murray and Muhammad tells the truth.Maria says Octavius lies. Marvin says Osborn tells the truth. Yoland says Delbert lies. Leda says exactly two of Alejandro, Othello and Mariana tell the truth. Sal says either exactly two of Kristian, Marvin and Osborn tell the truth or none of them. Oakley says Melinda lies. McKenzie says Orly lies. Nicole says Odin lies. Mark says Octavia lies. Odin says Nicole lies. Vina says Conception tells the truth. Muhammad says Murray lies. Orlando says Orla lies. Oakleigh says Olaf tells the truth. Ka says either exactly one of Yoland, Oakleigh and Olaf tells the truth, or all three of them. Odile says Miriam lies. Delbert says exactly one of Phoebe, Orly and McKenzie tells the truth.Osborn says Marvin tells the truth. Orion says Morgan tells the truth. Conception says only one of Antwan, Marina and Oakley lies. Jamey says Leda tells the truth. Jaymie says exactly two of Sal, Odin and Nicole tells the truth. Antwan says exactly one of Jaymie, Miriam and Odile tells the truth.Mackenzie says Mona tells the truth. Bahar says Orhan lies. Alejandro says only one of Lorine, Ollie and Bahar tells the truth. Audrie says exactly one of Vina, Orlando and Orla tell the truth. Do Alejandro, Willian, and Jamey tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}