{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ca3446e2-c3b8-5d79-9702-d4e01c5f74b1","task_key":"bbeh~5fweb~5fof~5flies--full--ca3446e2-c3b8-5d79-9702-d4e01c5f74b1","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the theater says the person at the ice skating rink tells the truth. The person at the gym tells the truth. Beatriz is at the gym. The person at the ice skating rink thinks their friend is lying. Grace is at the campground. Hiroshi is at the theater. Emily is at the farm. The person at the campground says the person at the observatory lies. The person at the botanical garden tells the truth. The person at the cafe says the person at the campground lies. The person at the farm lies. Priya is at the park. Maya is at the library. The person at the ice skating rink says the person at the city hall tells the truth. Charlie is at the cafe. The person at the park tells the truth. The person at the ice skating rink saw a firetruck. The person at the beach says the person at the theater lies. Nadia is at the ice skating rink. Ethan is at the observatory. The person at the campground lies. Max is at the museum. Ayaan is at the hotel. Jake is at the city hall. Jaxon is at the skate park. Luna is at the beach. Kehinde is at the train station. The person at the campground saw a firetruck. The person at the museum says the person at the theater lies. Olivia is at the botanical garden. The person at the theater says the person at the train station lies. The person at the skate park lies. The person at the ice skating rink says the person at the library tells the truth. The person at the ice skating rink says the person at the campground tells the truth. The person at the hotel says the person at the ice skating rink lies. Does the person at the theater tell the truth? Does the person at the ice skating rink tell the truth? Does the person at the campground tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}