{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9baf18c2-9f5b-58bd-af01-3a6f04bc3c3c","task_key":"default--validation--35117","task_revision_id":"1","upstream_id":"35117","short_description":"HellaSwag validation 35117","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to clean a couch [title] Determine the type of fabric your couch is made out of. [step] Sofas can be made of anything from cotton to leather, and knowing exactly what fabric you're working with is important for knowing how to clean it. Check the tag on the bottom of your sofa to determine the type of cloth.\",\"endings\":\"[\\\"[substeps] For some commercial sofas, you'll need to consult with a presales person or the same person who made them. For one, there may be rough patches of upholstery on the back and side of the sofa.\\\", \\\"[substeps] Fabric with lug nuts usually is not the material you'll want to use. If you have no idea, you can check the tag on a cheap stainless steel couch you don't know if it's made of leather or woven.\\\", \\\"The tag should have detailing on it indicating how old the material is, and bring it out to the vet if you're worried about your sofa's current style of covering. [title] Store vinyl and other materials in a clean bag or zip-close bag.\\\", 'The tag should give a label of \\\" w, \\\" \\\" s, \\\" \\\" ws, \\\" \\\" x, \\\" or \\\" o. \\\" [substeps] A tag with the letter \\\" w \\\" or \\\" ws \\\" means that your couch is washable with a water-based soap.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}