{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"daa13e7a-23b9-586d-bdb4-b1c3ee4c736b","task_key":"default--validation--13613","task_revision_id":"1","upstream_id":"13613","short_description":"HellaSwag validation 13613","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to do a sitting abductor exercise [title] Sit in a chair with your knees bent. [step] You want the chair to remain firmly in one place. Your feet should be planted on the floor, with your knees bent at ninety degrees.\",\"endings\":\"['Your knees should be touching one another, with the soles of both feet hanging off the side of the seat. [substeps] If you are doing the sit abductor exercise standing, make sure your feet are firmly together.', \\\"[substeps] Make sure the chair is not wobbly, and doesn't rotate or have wheels (e.g. an office chair) if you are at the gym, you can sit on a weightlifting bench.\\\", 'Your legs should be extended in front of you. Raise your arms above your head, as if you are praying.', 'Your left arm and right leg should be resting on the chair. [substeps] The seat should extend past your buttocks.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}