{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0ef39646-6ab4-563c-a6d1-0470a1d82508","task_key":"default--validation--18285","task_revision_id":"1","upstream_id":"18285","short_description":"HellaSwag validation 18285","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to make puffy paint [title] Combine the glue and shaving cream. [step] Add equal parts of white glue and shaving cream to a large bowl. Mix the two together until they are fully combined in a thick cream.\",\"endings\":\"['Spread the mixture over all surfaces of the canvas. [substeps] You will be creating the puffy paint in the shower, so mix the two at a time to ensure an even coverage.', 'Apply this mixture to the puffy paint. [substeps] You can also use a gel; for this recipe, use one of the products described in the section below.', \\\"[substeps] Be sure to use a shaving foam, not shaving gel. If you're concerned about the paint sticking to your spoons, you can use chopsticks, popsicle sticks, or even straws to mix the glue and shaving cream together.\\\", '[substeps] When making your own puffer paint, follow the instructions on the package. Mix the two together until they are wet.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}