{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d013a6b1-a185-5fd2-925b-1a7ca5c0144a","task_key":"bbeh~5fweb~5fof~5flies--full--d013a6b1-a185-5fd2-925b-1a7ca5c0144a","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Uri says Osborne lies. Odile says Oona tells the truth. Mehran says Oran tells the truth. Jerry says Ka tells the truth. Shenna says Shalonda tells the truth. Raymond says Shenna lies. Sal says only one of Raymond, Odell and Ozzy lies. Alejandro lies. Mackenzie says Marvin lies. Mandy says Oleg lies. Conception says only one of Sal, Oscar and Miller lies. Oleg says Mandy lies. Oden says Merlin tells the truth. Miller says Lalit lies. Maya says Merlin lies. Michael says Octavia lies. Marshall says Oona lies. Michelle says Oran lies. Maybelle says McKenzie lies. Amberly says exactly one of Ryan, Michael and Octavia tells the truth.Fletcher says only one of Audrie, Mehran and Michelle lies. Ka says Fletcher, Odile and Marshall all tell the truth. Oscar says Lalit tells the truth. Molly says Marvin tells the truth. Octavia says Michael lies. Shalonda says exactly one of Amberly, Osborne and Uri tell the truth. Sima says Lorine, Molly and Mackenzie all tell the truth. Jim says exactly two of Conception, Misty and Maximilian tell the truth. Kandi says only one of Sima, Oden and Maya lies. Mariana says Orlando tells the truth. Maximilian says Misty lies. Osborne says Uri lies. McKenzie says Maybelle lies. Misty says Maximilian lies. Audrie says exactly one of Alejandro, McKenzie and Maybelle tells the truth.Ozzy says Othello lies. Odell says Othello tells the truth. Orlando says Mariana tells the truth. Ryan says either exactly one of Kandi, Orlando and Mariana tells the truth, or all three of them. Lorine says exactly one of Jerry, Mandy and Oleg tells the truth.Do Shalonda, Conception, and Jim tell the truth? Your answer should be a list of three words separated by commas. Each of them must be either yes (if the person tells the truth), no (if the person lies), or unknown (if the person's truth cannot be determined). For example, <unknown, no, yes> or <yes, yes, yes>.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}