{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"335bc582-ffd9-5d54-aa50-d98e23d3a16b","task_key":"bbeh~5fweb~5fof~5flies--full--335bc582-ffd9-5d54-aa50-d98e23d3a16b","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the cafe saw a firetruck. Soren is at the school. Devika is at the gym. The person at the bookstore says the person at the planetarium tells the truth. Quan is at the cafe. Liam is at the farm. The person at the gym lies. The person at the movie theater lies. The person at the zoo thinks their friend is lying. Jaxon is at the train station. Elowen is at the planetarium. The person at the library says the person at the airport lies. The person at the zoo says the person at the park lies. The person at the museum says the person at the cafe lies. The person at the art gallery says the person at the airport lies. The person at the zoo says the person at the bookstore lies. Ayaan is at the bookstore. The person at the bookstore says the person at the cafe tells the truth. Charlie is at the vineyard. Max is at the library. The person at the bookstore thinks their neighbor is telling the truth. The person at the vineyard says the person at the zoo lies. Luna is at the airport. The person at the farm says the person at the airport tells the truth. Theo is at the park. The person at the school lies. Priya is at the art gallery. Tala is at the movie theater. Anika is at the museum. Emily is at the city hall. The person at the cafe tells the truth. Ethan is at the zoo. The person at the cafe says the person at the city hall tells the truth. The person at the airport says the person at the zoo tells the truth. The person at the train station lies. Does the person at the airport tell the truth? Does the person at the zoo tell the truth? Does the person at the bookstore tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}