{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8122b854-2792-55b6-bd0b-f125b4b9c365","task_key":"bbeh~5fweb~5fof~5flies--full--8122b854-2792-55b6-bd0b-f125b4b9c365","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Miller says Oden lies. Kristian says exactly two of Dallas, Miranda and Michaela tells the truth. Kate says Marcus tells the truth. Natasha says Nishanth tells the truth. Maximilian says Melinda tells the truth. Crista says exactly one of Tamika, Miller and Oden tell the truth. Yoland says Alejandro, Mia and Disha all tell the truth. Virginia says Morgan lies. Miranda says Michaela lies. Morris says Marshall lies. Milton says Nishanth lies. Gwenn says only one of Sima, Orlando and Murphy lies. Mike says Orion lies. Dallas says Audrie, Natasha and Milton all tell the truth. Mia says Micah tells the truth. Maurice says Matthew lies. Vernell says Inga, Millicent and Osborne all tell the truth. Disha says Micah lies. Helene lies. Oden says Miller lies. Lorine says either all three of Jamey, Kate and Marcus tell the truth or only one of them. Murphy says Mariah lies. Osborne says Minnie lies. Millicent says Minnie tells the truth. Fletcher says exactly two of Leda, Mike and Orion tell the truth. Marcus says Kate tells the truth. Jamey says only one of Vernell, Muriel and Maurice tells the truth. Ryan says Gwenn tells the truth. Morgan says Virginia lies. Inga says only one of Crista, Maximilian and Orin tells the truth. Alejandro says exactly two of Jim, Virginia and Morgan tell the truth. Osborn says Marshall tells the truth. Muriel says Matthew tells the truth. Orion says Mike lies. Michaela says Miranda lies. Leda says only one of Kristian, Osborn and Morris tells the truth. Orlando says Mariah tells the truth. Tamika says Fletcher tells the truth. Sima says Yoland tells the truth. Orin says Melinda lies. Jim says Lorine tells the truth. Audrie says Helene lies. Do Jamey, Disha, and Ryan tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}