{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"af780f8d-decc-5768-8a4f-cc034fd4b05b","task_key":"default--validation--37721","task_revision_id":"1","upstream_id":"37721","short_description":"HellaSwag validation 37721","config":"default","split":"validation","body":"{\"activity_label\":\"Cars & Other Vehicles\",\"ctx\":\"[header] How to drive in the snow [title] Prepare your vehicle for driving. [step] Clear the windshield, side windows, headlights, and rear brake lights of all snow and ice with an ice scraper and brush before you drive off. Stop the car and clear the windows and lights as needed while driving during snowfall.\",\"endings\":\"['You can buy snowblower helmets or winter hats from most retail stores. For better access, get snow pants and boots.', \\\"Don't forget to clear snow off of your number plate, as it is a legal requirement for your number plate to be visible in many places. [title] Turn on the front and rear defroster to keep windows clear while you drive.\\\", 'This will help you if you are going uphill or downhill. [substeps] Use a snow scraper to remove snow from grooves in the top of the windshield and headlights.', '[title] Block all leaking fluids from your vehicle with plastic, ash, gravel, and paper cover. [step] This provides protection in case of rain or snow, but as a general rule you should also clear excess air into your vehicle.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}