{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1035c953-0330-5d8d-a8b1-a99f0ffd1cad","task_key":"default--validation--38145","task_revision_id":"1","upstream_id":"38145","short_description":"HellaSwag validation 38145","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to clean vinyl shutters [title] Put on rubber and cotton gloves. [step] To start cleaning your shutters, put on a pair of rubber gloves. Then, place a pair of cotton gloves over your rubber gloves.\",\"endings\":\"['You do not need to use duct tape, but make sure you are wearing the gloves when you do so, as duct tape can be slippery. [title] Wipe off any grime with a cloth.', '[substeps] Make sure the cotton gloves you use are clean. Cleaning shutters with dirty gloves will only create more mess.', \\\"[substeps] It is important to keep your gloves firm as they will protect your hands. You can find rubber or cotton gloves at your local home improvement or farmers' market.\\\", 'Only remove the rubber gloves if you are absolutely certain that the material is safe to use on vinyl shutters. [substeps] If you do not have gloves, you can use a disposable pair of rubber gloves instead.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}