{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"04827795-cba6-5f4b-8d45-5a467022eeea","task_key":"default--validation--49068","task_revision_id":"1","upstream_id":"49068","short_description":"HellaSwag validation 49068","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to set a digital watch [title] Contact the manufacturer or look up a manual for complicated watches. [step] If your watch is especially complicated and you don't have instructions, look up a digital manual online. Digital manuals can be found with an online keyword search for your watch brand and model.\",\"endings\":\"['[substeps] Some watches turn off automatically during the day. If yours is in the morning, switch it to the middle of the night.', \\\"Your watch may take you a while, or it may take a little while to get to a level where you won't experience any problem. [substeps] Digital watches come with online cameras and camera boxes.\\\", '[substeps] If a digital manual is impossible to find, look up and call the customer service number for your watch. Model and brand information are usually written or engraved on the back of the watch or somewhere on the watch face.', 'They are usually $15-$40. They cover the exact specs of your watch and include instructions to change them.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}