{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"89fba1ea-a310-5692-803b-e83b07176c9e","task_key":"default--validation--23699","task_revision_id":"1","upstream_id":"23699","short_description":"HellaSwag validation 23699","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to customize your shoes [title] Doodle on canvas shoes with permanent markers or fabric markers. [step] Make sure that the shoes are clean first, then start doodling. If you are worried about making mistakes, you can sketch your design out lightly in pencil first.\",\"endings\":\"['You can also stop by a local art supply store and buy an eraser. [substeps] Erasing will make your shoes look sloppy.', \\\"If your shoes don't look like a basic sketch, color in your design over the brush. Continue doodling until your foot is centered and nothing else looks out of place.\\\", 'You can use all one color, such as black, or many colors. If you are using multiple colors, put the lighter colors down first, then the darker ones, and finally your outlines.', '[substeps] Be sure to doodle your design in the simplest way possible, otherwise you could mess up your art. Doodle on the canvas no matter what type of shoe you are wearing.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}