{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ec8b4780-3c0d-52bd-9a16-718404418a32","task_key":"default--validation--49250","task_revision_id":"1","upstream_id":"49250","short_description":"HellaSwag validation 49250","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to storm proof your home [title] Focus on rooftops, gutters and downspouts. [step] The best way to prepare a home for a rainy season is to direct water away from it and make sure those runoff channels are solid. Check that the gutters are clear of debris and that the downspouts aren't clogged or loose.\",\"endings\":\"['[title] Keep a storm meter on every roof using an electrical system. [step] This meter will go under the roof and measure water pressure.', \\\"It's not uncommon for roofs to slide out of the gutters and come crashing in a storm. [substeps] If there is debris on a roof, make sure that you're able to haul it away with free airflow to free up the wind.\\\", \\\"Have your roof's gutters monitored for those, so you can plan accordingly. [substeps] Before you leave, do some test runs.\\\", 'Consider pre-purchasing a gutter helmet to avoid having to clean out the gutters often. This will ensure that the spouts remain clear and unclogged all season.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}