{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2522ad84-9a27-5d35-b6f1-8a881e3bfa34","task_key":"bbeh~5fweb~5fof~5flies--full--2522ad84-9a27-5d35-b6f1-8a881e3bfa34","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Raymond says either all three of Jamey, Matteo and Melinda lie, or two of them tell the truth. Octavio says Osborn lies. Micah says Mark lies. Olivia says Misty tells the truth. Ryan says exactly two of Raymond, Mina and Mike tell the truth. Fletcher says Vernell lies. Sal says exactly two of Alejandro, Mark and Micah tell the truth. Willian says Andree lies. Alexis says either all three of Ryan, Osian and Mustafa tell the truth or only one of them. Matteo says Melinda tells the truth. Osian says Mustafa tells the truth. Vernell tells the truth. Miles says Megan tells the truth. Mark says Micah lies. Mike says Mina lies. Otis says Millie tells the truth. Mina says Mike lies. Oakleigh says Oscar lies. Misty says Olivia tells the truth. Shalonda says Sal tells the truth. Tamika says exactly one of Fletcher, Oakleigh and Oscar tells the truth.Mario says Oleg tells the truth. Alejandro says exactly two of Willian, Ocean and Virginia tell the truth. Gwenn says exactly two of Shalonda, Octavio and Osborn tell the truth. Jamey says either exactly two of Conception, Otis and Millie tell the truth or none of them. Mustafa says Osian tells the truth. Teressa says Ka tells the truth. Conception says either all three of Gwenn, Misty and Olivia tell the truth or only one of them. Andree says either exactly one of Teressa, Miles and Megan tells the truth, or all three of them. Megan says Miles tells the truth. Oscar says Oakleigh lies. Melinda says Matteo tells the truth. Oleg says Mario tells the truth. Millie says Otis tells the truth. Ocean says Virginia lies. Ka says either exactly two of Tamika, Oleg and Mario tell the truth or none of them. Virginia says Ocean lies. Osborn says Octavio lies. Do Conception, Raymond, and Alexis tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}