{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fe835439-caaf-5d71-bb2a-8f4b8a01b5cf","task_key":"bbeh~5fweb~5fof~5flies--full--fe835439-caaf-5d71-bb2a-8f4b8a01b5cf","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Ka says only one of Christie, Miller and Madeleine tells the truth. Odette says Orly lies. Melody says Osvaldo lies. Fidel lies. Oswin says Nicole tells the truth. Vina says exactly two of Fletcher, Osvaldo and Melody tell the truth. Osbert says Misty lies. Osborne says Maria tells the truth. Elanor says exactly one of Fidel, Misty and Osbert tells the truth.Mercedes says Oran tells the truth. Rashida says exactly one of Alejandro, Orly and Odette tells the truth.Fletcher says exactly one of Jamey, Marlon and Mustafa tells the truth.Jamey says Rashida, Oswin and Mona all tell the truth. Marlon says Mustafa lies. Shaunda says either exactly two of Sal, Oran and Mercedes tell the truth or none of them. Oran says Mercedes tells the truth. Conception says Gwenn tells the truth. Gwenn says Elanor, Marvin and Max all tell the truth. Madeleine says Bahar lies. Marvin says Oprah tells the truth. Miller says Bahar tells the truth. Teressa says Conception tells the truth. Christie says exactly one of Helene, Madison and Oden tell the truth. Owen says Martha lies. Sima says Vina tells the truth. Mustafa says Marlon lies. Max says Oprah lies. Martha says Owen lies. Mona says Nicole lies. Oden says Madison lies. Sal says exactly two of Ka, Martha and Owen tell the truth. Madison says Oden lies. Orly says Odette lies. Helene says only one of Teressa, Osborne and Mia tells the truth. Misty says Osbert lies. Alejandro says Shaunda tells the truth. Osvaldo says Melody lies. Mia says Maria lies. Do Vina, Madeleine, and Sima tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}