{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2f895905-92c0-5792-8c66-892c280c52e3","task_key":"bbeh~5fweb~5fof~5flies--full--2f895905-92c0-5792-8c66-892c280c52e3","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Hiroshi is at the farm. The person at the school tells the truth. The person at the train station says the person at the airport tells the truth. Nia is at the train station. The person at the city hall lies. Max is at the bowling alley. Quan is at the city hall. Theo is at the planetarium. The person at the museum tells the truth. The person at the observatory says the person at the airport lies. The person at the airport says the person at the bowling alley tells the truth. The person at the farm says the person at the barbershop lies. Ryan is at the school. Farid is at the airport. The person at the farm says the person at the theater tells the truth. The person at the planetarium says the person at the city hall lies. Tala is at the museum. Charlie is at the theater. The person at the barbershop says the person at the city hall tells the truth. The person at the observatory says the person at the shopping mall lies. The person at the airport says the person at the farm tells the truth. The person at the beach says the person at the city hall lies. Jaxon is at the beach. Liam is at the barbershop. The person at the airport says the person at the movie theater tells the truth. Kehinde is at the shopping mall. Ethan is at the movie theater. Mateo is at the observatory. Zara is at the vineyard. The person at the barbershop says the person at the vineyard lies. Does the person at the observatory tell the truth? Does the person at the airport tell the truth? Does the person at the farm tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}