{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa8de626-affb-53b8-bd6b-0fa73a86ea01","task_key":"default--validation--32359","task_revision_id":"1","upstream_id":"32359","short_description":"HellaSwag validation 32359","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to climb a ladder safely [title] Raise the ladder. [step] Hold the ladder so that each foot is planted firmly against the ground. Lift the top of the ladder and walk towards the wall.\",\"endings\":\"[\\\"[substeps] You will need to judge the height and position of your ladder's rungs by counting them, not counting the jump height-they will be between 22 and 40 feet (0.91 and 1.9 m). [title] Put your feet on the vertical metal rungs.\\\", 'If you have an extension ladder, raise the rungs, either by hand or by using the rope and pulley wheel. [title] Set the ladder at a 75 ° angle with the ground.', 'Be careful not to call out something like \\\" hold this, hold this! \\\" or \\\" no, hold here! \\\" being too hasty can cause you to fall. Stay in front of the wall until you reach the height of the ladder.', \\\"Don't forget to hold on to the side of the ladder that is facing your door. You will have to reach to lower the ladder onto the ground.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}