{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"72450064-720a-5d96-a93a-4701054e8c14","task_key":"bbeh~5fweb~5fof~5flies--full--72450064-720a-5d96-a93a-4701054e8c14","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the hotel thinks their friend is lying. The person at the bookstore lies. Elowen is at the city hall. The person at the observatory says the person at the restaurant tells the truth. Tala is at the restaurant. Kehinde is at the gym. The person at the bowling alley tells the truth. Jake is at the hotel. Ethan is at the aquarium. Soren is at the observatory. The person at the hotel says the person at the aquarium lies. Mateo is at the bookstore. Max is at the skate park. The person at the barbershop tells the truth. The person at the skate park says the person at the gym tells the truth. Devika is at the barbershop. Nia is at the cafe. The person at the school says the person at the botanical garden lies. Isabella is at the shopping mall. Farid is at the movie theater. The person at the gym says the person at the theater lies. The person at the gym says the person at the botanical garden lies. The person at the hotel says the person at the gym tells the truth. Liam is at the museum. The person at the observatory says the person at the hotel tells the truth. The person at the city hall says the person at the botanical garden tells the truth. The person at the cafe says the person at the hotel tells the truth. Grace is at the botanical garden. Zara is at the bowling alley. The person at the botanical garden tells the truth. The person at the museum lies. The person at the hotel says the person at the shopping mall lies. Maya is at the theater. Anika is at the school. The person at the movie theater says the person at the hotel lies. Does the person at the observatory tell the truth? Does the person at the hotel tell the truth? Does the person at the gym tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}