{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"204ed9d5-4c5d-5043-9284-f619fed2fca5","task_key":"bbeh~5fweb~5fof~5flies--full--204ed9d5-4c5d-5043-9284-f619fed2fca5","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Melanie says Oriana tells the truth. Melody says Maybelle lies. Oakleigh says Nicole tells the truth. Sal says Andree tells the truth. Jaymie says only one of Shalonda, Disha and Melody lies. Sherrie says either exactly one of Willian, Oriana and Melanie tells the truth, or all three of them. Amberly says Sherrie lies. Oscar says Mariah lies. Miles says Millicent lies. Odysseus says Marlon lies. Osmond says Orla tells the truth. Disha says Maybelle tells the truth. Inga says exactly two of Amberly, Millicent and Miles tell the truth. Fletcher says exactly one of Lorine, Oscar and Mariah tell the truth. Oriana says Melanie tells the truth. Octavio says Michaela lies. Maynard says Olaf lies. Millicent says Miles lies. Orla says Osmond tells the truth. Lorine says Shaunda tells the truth. Ovid says Michaela tells the truth. Shalonda says Kandi lies. Shaunda says only one of Leda, Oakleigh and Mike lies. Mariah says Oscar lies. Mike says Nicole lies. Oakley says Mavis lies. Leda says exactly two of Jim, Marlon and Odysseus tell the truth. Marlon says Odysseus lies. Kandi says Inga, Ovid and Octavio all tell the truth. Willian says either exactly two of Sal, Orla and Osmond tell the truth or none of them. Mavis says Oakley lies. Vernell says exactly one of Jaymie, Olaf and Maynard tell the truth. Jim tells the truth. Andree says exactly one of Fletcher, Mavis and Oakley tells the truth.Olaf says Maynard lies. Do Inga, Orla, and Vernell tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}