{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"833727b0-704f-5948-a109-c113f51f65f2","task_key":"bbeh~5fweb~5fof~5flies--full--833727b0-704f-5948-a109-c113f51f65f2","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the farm thinks their friend is lying. Tala is at the vineyard. The person at the farm says the person at the museum tells the truth. Nia is at the campground. The person at the observatory says the person at the gym lies. The person at the vineyard says the person at the observatory lies. The person at the zoo tells the truth. The person at the cafe says the person at the amusement park lies. Ayaan is at the amusement park. Liam is at the observatory. The person at the library lies. Grace is at the farm. The person at the farm says the person at the vineyard tells the truth. The person at the movie theater says the person at the zoo tells the truth. The person at the gym says the person at the amusement park lies. The person at the gym says the person at the restaurant tells the truth. Luna is at the museum. Jaxon is at the bookstore. The person at the bookstore tells the truth. Isabella is at the botanical garden. The person at the gym says the person at the botanical garden tells the truth. Anika is at the ice skating rink. Emily is at the zoo. Theo is at the library. Mateo is at the gym. Ethan is at the restaurant. The person at the amusement park says the person at the zoo lies. Nadia is at the cafe. Elowen is at the movie theater. The person at the amusement park says the person at the campground tells the truth. The person at the ice skating rink tells the truth. Does the person at the farm tell the truth? Does the person at the vineyard tell the truth? Does the person at the observatory tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}