{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aba75bb3-4620-5b92-9c77-f190403f6911","task_key":"default--validation--31324","task_revision_id":"1","upstream_id":"31324","short_description":"HellaSwag validation 31324","config":"default","split":"validation","body":"{\"activity_label\":\"Cars & Other Vehicles\",\"ctx\":\"[header] How to install your premounted winter tires [title] Gather the materials needed for the job. [title] Remove all four wheel covers (if equipped) by sliding the screwdriver between the wheel and cover. [step] Gently pry the cover off.\",\"endings\":\"['Once the cover comes off, use a razor blade to slice a 2 in (5.1 cm) file through the side of cover. [substeps] This file is designed for getting rid of old tires because you will be using it for storage instead of completing your work.', '[title] Remove the original bolts that attach the tire cover. [step] Mine the bolts in the clips at the wheels.', 'Turn the vehicle over, securing the cover with screws. [substeps] More commonly, winter tires come with chrome rims.', '[title] With a standard driver and socket, loosen slightly all wheel nuts on all four wheels. [step] Never use a torque wrench to break loose fasteners.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}