{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64b0edfc-ceac-5942-951e-77b60fa912c1","task_key":"default--validation--18304","task_revision_id":"1","upstream_id":"18304","short_description":"HellaSwag validation 18304","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to straight lace shoes [title] Insert shoelace into the first eyelets of your first shoe. [step] Point the toe of the shoe away from you. The holes farthest from you will be called the first eyelets, and will successively count up from there.\",\"endings\":\"['[title] Push the center eyelet on the top of the shoe out of the way. [step] This eyelet is where your fourth shoe will be folded up.', '[substeps] Begin with the top eyelet and move to the bottom eyelet. The laces and binding agent should be equidistant from each other and oriented in a way that remains loose.', '[title] Thread the lace between first eyelets. [step] Slip the lace through both eyelets in the first eyelet closest to you.', 'With the shoelace on the outside of the shoe, insert the ends down into the first eyelets on both sides. [title] Make sure the shoelace ends are even.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}