{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0bc4e70c-4f69-54e7-a373-4f5c98ccb0df","task_key":"default--validation--359","task_revision_id":"1","upstream_id":"359","short_description":"HellaSwag validation 359","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to can pears [title] Wash the jars and lids. [step] Use dish detergent and hot water to scrub the jars and lids, especially if they were previously used. Make sure to thoroughly rinse them with running water to remove any soap.\",\"endings\":\"['[title] Place the jars and lids on the tree in their upright position. [step] Place one jar at a time on the back of the tree, and position the jars 2/3 of the way apart from each other.', '[title] Sanitize and heat the lids. [step] Set a small pot of water to a boil over medium to medium-high heat.', \\\"[substeps] Scrub the lids gently so they aren't too stiff. Rub at them lightly with a cloth until they are clean.\\\", '[substeps] If possible, flush jars under hot to lukewarm water. Cool the jars for thirty seconds after washing them.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}