{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"46779a6f-84f7-56ce-bea9-47e387984e6f","task_key":"default--validation--11520","task_revision_id":"1","upstream_id":"11520","short_description":"HellaSwag validation 11520","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to scotchgard shoes [title] Take your shoes to a well-ventilated area. [step] Scotchgard fabric protector is high in fumes, which you want to avoid breathing for too long. Going outside to spray your shoes is the best option if the weather is good.\",\"endings\":\"['[substeps] Starchy material thalights your shoes easily. You can buy it at any major retailer that sells school supplies, such as walmart.', \\\"[title] Moisten the sole of your shoe. [step] You don't have to do this; just wet it.\\\", 'Otherwise, go into a garage or open the windows in the room. [substeps] Keep your kids and pets away while you spray your shoes.', '[substeps] Do not spray your shoes with water or the bleaching could damage them. [title] Lay your shoes out on a towel.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}