{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"208ae57b-f1f0-5ef0-9d8c-13bff3418169","task_key":"bbeh~5fweb~5fof~5flies--full--208ae57b-f1f0-5ef0-9d8c-13bff3418169","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Andree says exactly two of Shenna, Miles and Mavis tell the truth. Mackenzie says Virginia tells the truth. Milton says Madison lies. Millie says Michaela tells the truth. Odin says Merlin tells the truth. Ofelia says Uri tells the truth. Sherrie says Ka, Muriel and Maynard all tell the truth. Tamika says either exactly two of Andree, Mustafa and Osric tell the truth or none of them. Muriel says Milo tells the truth. Octavia says Mike lies. Oswald says Oran tells the truth. Mustafa says Osric tells the truth. Inga says exactly one of Helene, Mike and Octavia tell the truth. Micah says Osbert tells the truth. Jaymie says Leda tells the truth. Osric says Mustafa tells the truth. Mark says Mariah lies. Oswin says Owen tells the truth. Ka says exactly two of Fidel, Mariah and Mark tell the truth. Fidel says only one of Shaunda, Minerva and Osvaldo tells the truth. Vernell says either exactly one of Jamey, Bahar and Oleg tells the truth, or all three of them. Shenna says exactly one of Sherrie, Madison and Milton tells the truth.Mavis says Miles lies. Uri says Ofelia tells the truth. Mercedes says Osbert lies. Maynard says Milo lies. Mike says Octavia lies. Miles says Mavis lies. Michaela says Millie tells the truth. Merlin says Odin tells the truth. Christie says only one of Inga, Micah and Mercedes tells the truth. Jamey says either exactly one of Conception, Merlin and Odin tells the truth, or all three of them. Owen says Oswin tells the truth. Helene says either all three of Raymond, Millie and Michaela lie, or two of them tell the truth. Odile says Oran lies. Raymond says either exactly one of Kristian, Ofelia and Uri tells the truth, or all three of them. Virginia says Mackenzie tells the truth. Shaunda says Jaymie tells the truth. Mariah says Mark lies. Bahar says Oleg tells the truth. Oleg says Bahar tells the truth. Vina says either all three of Vernell, Owen and Oswin lie, or two of them tell the truth. Osvaldo says Orla lies. Conception says Christie, Oswald and Odile all tell the truth. Kristian lies. Madison says Milton lies. Leda says either exactly two of Vina, Mackenzie and Virginia tell the truth or none of them. Minerva says Orla tells the truth. Do Jaymie, Odin, and Tamika tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}