{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8c4c8ff4-3a87-5366-9c35-8d9e603d04d5","task_key":"bbeh~5fweb~5fof~5flies--full--8c4c8ff4-3a87-5366-9c35-8d9e603d04d5","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. Luna is at the barbershop. The person at the skate park tells the truth. Nadia is at the skate park. The person at the barbershop tells the truth. The person at the shopping mall says the person at the school lies. The person at the bookstore says the person at the shopping mall tells the truth. Nia is at the cafe. Isabella is at the school. Ayaan is at the amusement park. Quan is at the zoo. Maya is at the farm. The person at the amusement park tells the truth. The person at the bookstore says the person at the airport tells the truth. Ryan is at the bookstore. Jaxon is at the shopping mall. The person at the cafe tells the truth. The person at the art gallery says the person at the farm tells the truth. Soren is at the art gallery. The person at the shopping mall says the person at the farm lies. The person at the farm says the person at the amusement park lies. Olivia is at the airport. The person at the zoo tells the truth. Does the person at the bookstore tell the truth? Does the person at the shopping mall tell the truth? Does the person at the farm tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}