{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"22ef7728-ab86-59c2-8194-8fba27499da5","task_key":"bbeh~5fweb~5fof~5flies--full--22ef7728-ab86-59c2-8194-8fba27499da5","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Maximilian says Oliver lies. Kandi says exactly one of Sima, Odell and Othello tell the truth. Fletcher says exactly two of Rashida, John and Margaret tells the truth. Mallory says Mickey lies. Ryan says either all three of Delfina, Kate and Malcolm lie, or two of them tell the truth. Maybelle says Melissa tells the truth. Fidel says only one of Ka, Bahar and Maximilian lies. Vernell says either exactly one of Fletcher, Marlon and Ofelia tells the truth, or all three of them. Tamika tells the truth. Delfina says either exactly two of Tamika, Sanket and Mackenzie tell the truth or none of them. Ka says exactly two of Kandi, Mallory and Mickey tells the truth. Sanket says Mackenzie tells the truth. Sima says Raymond tells the truth. Margaret says John lies. Kristian says Vernell tells the truth. Bernita says exactly two of Gwenn, Ollie and Osric tells the truth. Odell says Othello lies. Melissa says Maybelle tells the truth. Marvin says Onyx tells the truth. Osric says Ollie lies. Mackenzie says Sanket tells the truth. Gwenn says either exactly one of Kristian, Maybelle and Melissa tells the truth, or all three of them. Crista says Fidel lies. Elanor says Helene tells the truth. Kate says Malcolm tells the truth. Marlon says Ofelia tells the truth. Raymond says Elanor lies. Onyx says Marvin tells the truth. Rashida says either exactly two of Ryan, Marvin and Onyx tell the truth or none of them. Othello says Odell lies. Ollie says Osric lies. Mickey says Mallory lies. Malcolm says Kate tells the truth. John says Margaret lies. Ofelia says Marlon tells the truth. Bahar says Oliver tells the truth. Helene says Bernita tells the truth. Do Elanor, Kandi, and Crista tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}