{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"32fe451a-eabe-5ed5-b670-da9cd69ba7dc","task_key":"default--validation--8061","task_revision_id":"1","upstream_id":"8061","short_description":"HellaSwag validation 8061","config":"default","split":"validation","body":"{\"activity_label\":\"Education and Communications\",\"ctx\":\"[header] How to write movie, music, and video game reviews [title] Watch, play, or listen repeatedly. [step] The first step in any review is to become as familiar as you can with the media product you are reviewing. Spend some time watching, listening, or playing.\",\"endings\":\"['Research shows how much of the last 40 million-year movie reviews were gained through immersion in the internet. [title] Explore the various audio sourcebooks and forums related to video game reviews.', 'Always understand what the product is about and get familiar with it. Read the reviews for any inaccuracies that you find.', '[substeps] For movies, you should plan on watching the film at least two or three times. The first time, just watch and enjoy as your normally would.', 'Try not to lean too heavily on that experience. Try to stay inconstant (or at least not get distracted).']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}