{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"76ea2ffe-88ab-5e9d-b496-25329e39ee50","task_key":"bbeh~5fweb~5fof~5flies--full--76ea2ffe-88ab-5e9d-b496-25329e39ee50","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Minnie says Miller tells the truth. Miller says Minnie tells the truth. Ottoline says Micah lies. Mateo says Odile lies. Othello says Orin lies. Christie says Fletcher tells the truth. Morgan says Oona lies. Fletcher says Teressa, Oakleigh and Maxwell all tell the truth. Vina says either all three of Inga, Maverick and Mary tell the truth or only one of them. Delbert says exactly two of Vina, Othello and Orin tell the truth. Phoebe says only one of Christie, Madison and Ottoline tells the truth. Inga says either all three of Bernita, Minnie and Miller lie, or two of them tell the truth. Odile says Mateo lies. Alejandro says exactly two of Jim, Mateo and Odile tell the truth. Orin says Othello lies. Martha says Onyx lies. Mary says Maverick tells the truth. Jim says exactly two of Gwenn, Morgan and Oona tell the truth. Crista says either exactly two of Phoebe, Mike and Michael tell the truth or none of them. Shenna says Alejandro tells the truth. Maverick says Mary tells the truth. Michael says Mike tells the truth. Madison says Micah tells the truth. Ka says Crista lies. Mike says Michael tells the truth. Conception says Shenna tells the truth. Teressa tells the truth. Oona says Morgan lies. Oakleigh says Bahar tells the truth. Maxwell says Bahar lies. Bernita says Ka lies. Onyx says Martha lies. Gwenn says exactly two of Delbert, Martha and Onyx tell the truth. Do Alejandro, Gwenn, and Conception tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}