{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3c01636f-9492-5dfa-b7fa-fec57e66a8c8","task_key":"bbeh~5fweb~5fof~5flies--full--3c01636f-9492-5dfa-b7fa-fec57e66a8c8","task_revision_id":"1","upstream_id":"","short_description":"In this question, assume each person either always tells the truth or always…","config":"bbeh_web_of_lies","split":"full","body":"{\"input\":\"In this question, assume each person either always tells the truth or always lies. The person at the campground says the person at the train station tells the truth. The person at the movie theater thinks their friend is lying. The person at the movie theater says the person at the campground tells the truth. Mateo is at the library. The person at the movie theater thinks their neighbor is telling the truth. Olivia is at the train station. The person at the vineyard says the person at the movie theater tells the truth. The person at the aquarium thinks their friend is lying. The person at the movie theater saw a firetruck. The person at the movie theater says the person at the cafe tells the truth. The person at the campground saw a firetruck. Beatriz is at the cafe. Grace is at the shopping mall. Kehinde is at the movie theater. The person at the campground tells the truth. Charlie is at the campground. The person at the aquarium says the person at the movie theater lies. Zara is at the aquarium. The person at the movie theater says the person at the shopping mall tells the truth. Maya is at the vineyard. The person at the aquarium says the person at the library lies. Does the person at the aquarium tell the truth? Does the person at the movie theater tell the truth? Does the person at the campground tell the truth? Your answer should be a list of three words, yes or no (for example, yes, no, yes).\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}