{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1ed5838f-7e74-5558-a9d4-365ad9e44135","task_key":"bbeh~5fweb~5fof~5flies--full--1ed5838f-7e74-5558-a9d4-365ad9e44135","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the farm lies. Soren is at the zoo. The person at the vineyard says the person at the library lies. The person at the library says the person at the ice skating rink lies. The person at the ice skating rink tells the truth. The person at the ice skating rink saw a firetruck. Isabella is at the movie theater. The person at the movie theater tells the truth. The person at the vineyard saw a firetruck. Luna is at the cafe. The person at the planetarium says the person at the vineyard tells the truth. The person at the zoo says the person at the cafe tells the truth. Maya is at the planetarium. Ethan is at the farm. The person at the zoo says the person at the skate park lies. Max is at the beach. The person at the skate park says the person at the beach tells the truth. The person at the beach says the person at the planetarium lies. Kehinde is at the airport. The person at the airport lies. Charlie is at the ice skating rink. The person at the ice skating rink thinks their neighbor is telling the truth. Elowen is at the vineyard. Quan is at the skate park. Tala is at the library. Does the person at the zoo tell the truth? Does the person at the skate park tell the truth? Does the person at the beach tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}