{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31697ea6-eb5a-5941-9c56-a8a3b3838a75","task_key":"bbeh~5fweb~5fof~5flies--full--31697ea6-eb5a-5941-9c56-a8a3b3838a75","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Sima says Jim lies. Raymond says exactly one of Fletcher, Milton and Mitchell tell the truth. Oden says Michelle tells the truth. Jaymie says Dallas lies. Mavis says Maximilian tells the truth. Jim says Crista tells the truth. Milton says Mitchell lies. Orlando says Octavio tells the truth. Maxwell says Orson lies. Michelle says Oden tells the truth. Kristian says Sima lies. Vina says either exactly one of Raymond, Mavis and Maximilian tells the truth, or all three of them. Shalonda says Audrie tells the truth. Olympia says Miranda tells the truth. Delbert says Shalonda tells the truth. Bernita says either all three of Sal, Oden and Michelle tell the truth or only one of them. Marty says Marvin tells the truth. Sal says either all three of Phoebe, Orlando and Octavio tell the truth or only one of them. Alejandro says exactly one of Kristian, Orson and Maxwell tell the truth. Orson says Maxwell lies. Mitchell says Milton lies. Oscar says Otis lies. Otis says Oscar lies. Fidel says exactly two of Bernita, Otis and Oscar tell the truth. Dallas says exactly two of Delbert, Miriam and Miguel tells the truth. Miguel says Miriam lies. Maximilian says Mavis tells the truth. Miriam says Miguel lies. Fletcher says only one of Fidel, Marty and Owen lies. Crista says either all three of Vina, Olympia and Miranda lie, or two of them tell the truth. Phoebe says Jaymie tells the truth. Owen says Marvin lies. Oakley says Ozzy tells the truth. Ozzy says Oakley tells the truth. Octavio says Orlando tells the truth. Miranda says Olympia tells the truth. Conception says either all three of Alejandro, Ozzy and Oakley lie, or two of them tell the truth. Audrie lies. Do Alejandro, Jim, and Conception tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}