{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae8261da-e1d2-5a25-88b4-e0a9e5e963a6","task_key":"bbeh~5fweb~5fof~5flies--full--ae8261da-e1d2-5a25-88b4-e0a9e5e963a6","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Mario says Mavis tells the truth. Octavia says Malcolm tells the truth. Delfina says only one of Dallas, Octavio and Mickey lies. Milo says Moses tells the truth. Kristian says exactly two of Kandi, Maya and Morris tells the truth. Jamey says either exactly one of Kristian, Malcolm and Octavia tells the truth, or all three of them. Mickey says Micah lies. Malcolm says Octavia tells the truth. Raymond says either all three of Shaunda, Milo and Moses tell the truth or only one of them. Ryan says Jamey tells the truth. Ollie says Millie tells the truth. Oakleigh says Melissa tells the truth. Mia says Uri lies. Melinda says Maximilian tells the truth. Millie says Ollie tells the truth. Kandi lies. Osborne says Mavis lies. Melissa says Oakleigh tells the truth. Marina says Odette lies. Octavio says Micah tells the truth. Moses says Milo tells the truth. Dallas says Tamika lies. Maya says Morris lies. Shaunda says Elanor lies. Gwenn says only one of Delfina, Oden and Marina tells the truth. Sal says either exactly one of Willian, Ollie and Millie tells the truth, or all three of them. Elanor says only one of Delbert, Mario and Osborne lies. Delbert says either exactly two of Gwenn, Melissa and Oakleigh tell the truth or none of them. Oden says Odette tells the truth. Oscar says Maxim tells the truth. Maximilian says Melinda tells the truth. Uri says Mia lies. Willian says exactly one of Raymond, Mia and Uri tell the truth. Jim says either exactly one of Ryan, Maxim and Oscar tells the truth, or all three of them. Morris says Maya lies. Maxim says Oscar tells the truth. Tamika says either all three of Jim, Maximilian and Melinda lie, or two of them tell the truth. Do Shaunda, Marina, and Sal tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}