{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0a532e67-abb6-5eea-bc32-1c6e5b441399","task_key":"bbeh~5fweb~5fof~5flies--full--0a532e67-abb6-5eea-bc32-1c6e5b441399","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the barbershop says the person at the zoo tells the truth. The person at the bookstore says the person at the library lies. Zara is at the bookstore. The person at the farm lies. Ryan is at the observatory. Ethan is at the botanical garden. Priya is at the vineyard. The person at the observatory says the person at the campground tells the truth. The person at the museum says the person at the botanical garden lies. Grace is at the museum. Farid is at the gym. The person at the vineyard says the person at the farm lies. The person at the library says the person at the museum lies. The person at the campground says the person at the vineyard tells the truth. Olivia is at the campground. The person at the gym says the person at the zoo lies. Devika is at the barbershop. The person at the farm saw a firetruck. The person at the zoo says the person at the bookstore tells the truth. Hiroshi is at the farm. The person at the gym thinks their friend is lying. The person at the museum says the person at the observatory tells the truth. Emily is at the library. The person at the bookstore saw a firetruck. Isabella is at the zoo. Does the person at the gym tell the truth? Does the person at the zoo tell the truth? Does the person at the bookstore tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}