{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f3efdad1-af61-5f0e-8ec9-712dd0cf2c00","task_key":"bbeh~5fweb~5fof~5flies--full--f3efdad1-af61-5f0e-8ec9-712dd0cf2c00","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Emily is at the shopping mall. The person at the skate park says the person at the shopping mall lies. The person at the botanical garden says the person at the school tells the truth. Mateo is at the cafe. The person at the hotel tells the truth. Luna is at the zoo. Kehinde is at the observatory. Max is at the airport. The person at the art gallery says the person at the train station tells the truth. The person at the train station says the person at the bowling alley lies. The person at the campground says the person at the amusement park lies. The person at the gym says the person at the observatory lies. The person at the airport tells the truth. Jake is at the aquarium. Ryan is at the gym. The person at the cafe says the person at the planetarium lies. The person at the library says the person at the planetarium lies. Nia is at the bookstore. The person at the bookstore says the person at the skate park lies. The person at the gym thinks their neighbor is telling the truth. The person at the gym says the person at the train station lies. Ethan is at the train station. The person at the planetarium lies. The person at the shopping mall says the person at the gym tells the truth. The person at the campground says the person at the skate park lies. Farid is at the library. Soren is at the hotel. Olivia is at the art gallery. Beatriz is at the planetarium. Liam is at the school. Charlie is at the botanical garden. Maya is at the skate park. The person at the zoo says the person at the shopping mall tells the truth. Grace is at the restaurant. The person at the campground says the person at the aquarium lies. The person at the skate park thinks their neighbor is telling the truth. Ayaan is at the bowling alley. The person at the botanical garden says the person at the planetarium tells the truth. The person at the train station says the person at the botanical garden tells the truth. Theo is at the amusement park. Jaxon is at the campground. The person at the restaurant lies. Does the person at the campground tell the truth? Does the person at the skate park tell the truth? Does the person at the shopping mall tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}