{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"374a9a29-6955-5dba-938a-36123405b4ae","task_key":"bbeh~5fweb~5fof~5flies--full--374a9a29-6955-5dba-938a-36123405b4ae","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Ovid says Michaela lies. Shaunda says either all three of Helene, Mina and Matteo tell the truth or only one of them. Meredith says Onyx tells the truth. Vernell says Dallas tells the truth. Matthew says Mia lies. Osric says Mercedes tells the truth. Onyx says Meredith tells the truth. Mariana says Meadow tells the truth. Mercedes says Osric tells the truth. Willian says either all three of Yoland, Mercedes and Osric lie, or two of them tell the truth. Bernita says either all three of Vernell, Mariana and Meadow lie, or two of them tell the truth. Maya says Megan tells the truth. Jim says Sima tells the truth. Mia says Matthew lies. Opal says Oliver tells the truth. Michaela says Ovid lies. Oliver says Opal tells the truth. Gwenn lies. Andree says exactly two of Ryan, Ovid and Michaela tell the truth. Millie says Mustafa tells the truth. Audrie says only one of Ka, Mohamed and Melvin tells the truth. Matteo says Mina tells the truth. Dallas says Audrie, Marty and Lalit all tell the truth. Lalit says Mickey lies. Miguel says Bahar tells the truth. Sima says Shaunda lies. Lorine says either exactly one of Tamika, Mustafa and Millie tells the truth, or all three of them. Ryan says Bernita tells the truth. Muriel says Marcus tells the truth. Ka says either exactly two of Crista, Meredith and Onyx tell the truth or none of them. Marty says Mickey tells the truth. Tamika says either exactly one of Jaymie, Opal and Oliver tells the truth, or all three of them. Bahar says Miguel tells the truth. Alejandro says Gwenn lies. Mustafa says Millie tells the truth. Jaymie says either exactly one of Jim, Marcus and Muriel tells the truth, or all three of them. Meadow says Mariana tells the truth. Helene says either all three of Willian, Maya and Megan tell the truth or only one of them. Megan says Maya tells the truth. Mina says Matteo tells the truth. Yoland says exactly one of Alejandro, Mia and Matthew tell the truth. Melvin says Osbert lies. Crista says either all three of Lorine, Miguel and Bahar lie, or two of them tell the truth. Marcus says Muriel tells the truth. Mohamed says Osbert tells the truth. Do Bernita, Onyx, and Andree tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}