{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"91605c54-5aa5-5353-8fcd-fd82d19119d5","task_key":"default--validation--25986","task_revision_id":"1","upstream_id":"25986","short_description":"HellaSwag validation 25986","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to start a hot tub [title] Shut off power to the hot tub. [step] While cleaning and getting your hot tub ready, you might accidentally turn it on. Running your hot tub without water or too little water can damage its pump and heater.\",\"endings\":\"[\\\"There are a variety of steps that you can take to shut off the hot tub : [substeps] Open a fan or step outside in an open area. If you are using a friend's hot tub, open the radiator.\\\", 'To prevent this from happening, set your hot tub fuse in your fuse box or electrical panel to \\\" off. \\\" [substeps] Depending on your home, the location of your fuse box or electrical panel may vary.', 'To avoid this, shut off the power to the hot tub, then turn the power back on. [title] Unplug the hot tub from the power source.', 'When you open and shut the hose on the hot tub, you should hear a click sound that indicates how to shut off the water. [substeps] Turn on the hot water valve to determine if it is turning on or off.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}