{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ed702fdb-a838-5b30-b4b0-d7caead061f0","task_key":"default--validation--46261--variant-a76c16e679cf","task_revision_id":"1","upstream_id":"46261--variant-a76c16e679cf","short_description":"HellaSwag validation 46261","config":"default","split":"validation","body":"{\"activity_label\":\"Cars & Other Vehicles\",\"ctx\":\"[header] How to track your gas mileage with fuelfrog [title] Go to fuelfrog to sign up for a free account. [title] Go to the next page. [step] It will take you to your fuelfrog dashboard.\",\"endings\":\"['[title] Near your dashboard is a button that says \\\" change fuel. \\\" click this button and enter your current fuel mileage.', 'This is where you can enter in the information on the website. It works best if you can enter two or three fill-ups.', \\\"[title] Go ahead and pay the upfront fee. [step] Don't worry, it's slightly higher than the online fees.\\\", '[title] Click on \\\" new step \\\" at the top right side of the dashboard. [title] Look where the steps on the dashboard have appeared.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}