{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6fe1d6b2-f8e6-5b5b-8ff9-5a9bf0ea1717","task_key":"default--validation--7290","task_revision_id":"1","upstream_id":"7290","short_description":"HellaSwag validation 7290","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to be a new york mets fan [title] Start watching mets games. [step] Pay attention to the whole game, don't drift in and out. Decide which player you like the most by watching the plays he makes on the field and his interviews after the game (their personality is important too! ).\",\"endings\":\"['[title] Perform cpr at the end of any matches. [step] Double check your id when the game is over when asked to perform the cpr.', 'It\\\\'s vital that you can play in front of friends and family, but in general you like to watch the games anyway, so it\\\\'s easy to become a fan. [substeps] Get familiar with the phrase \\\" mets \\\" and how it says \\\" mets rivalry \\\".', \\\"[title] Get into your favorite sports league and watch the mets games, but don't watch them with someone else. [step] If you see the mets team during the movie, vomit it.\\\", '[title] Learn about the mets history (they won the world series in 1969 and 1986, were created in 1962, etc) and research some team trivia. [step] This can help you hold your own in conversations with even lifelong die-hards.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}