{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"72600aaa-a39e-5373-b6cb-8ce1b52f5a8a","task_key":"default--validation--34942","task_revision_id":"1","upstream_id":"34942","short_description":"HellaSwag validation 34942","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to use hot stones [title] Find smooth, flat rocks. [step] Look for river rocks, which tend to be smooth and flat. You can also purchase soapstones, which retain heat well.\",\"endings\":\"['Try to find them from specialty stores, or online through ebay or amazon. [title] Wash the stones with hot water and antibacterial dish soap.', 'You can purchase rocks at hiking supply stores or online. Set the rock on a flat or bumpy surface, such as a log, and create a small indentation in the rock.', \\\"They are also hard to work with, so try to use flat rocks that are at least half as round as the rock you want. [substeps] Hot rocks won't hold their shape well if they aren't soft.\\\", 'You want ones with cracks and crevices that are not too deep so that you can heat the rocks easily. [title] Set stones in the river, on a flat surface.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}