{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"98992ce8-e725-5818-aa76-03ea3aa6636f","task_key":"default--validation--47214","task_revision_id":"1","upstream_id":"47214","short_description":"HellaSwag validation 47214","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to paint your toe nails [title] Remove your old polish. [step] The first thing to do is remove any polish that's already on your nails. Soak a cotton ball with nail polish remover and run it over your nails until all the polish is gone.\",\"endings\":\"['[substeps] A q-tip can help with hard-to-reach edges. When choosing nail polish remover, be aware that those with acetone are more effective, but also more likely to damage your hands.', 'Then, remove all of your old polish and wash your hands. Next, use your fingertips to scrub and polish the rest of your nails, being gentle and never scratching.', 'Peel the acrylic paint from the cotton ball and then pat it dry with clean paper towels. [title] Choose a pair of short, pointed nails.', '[title] Wait at least one hour and repeat until your polish is completely gone. [step] It should come off in one application, so just keep going.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}