{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9a318459-3532-5b9d-b117-db851938d6b9","task_key":"default--validation--15859","task_revision_id":"1","upstream_id":"15859","short_description":"HellaSwag validation 15859","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to paint basement stairs [title] Remove any carpet or overlaid material from your basement stairs. [step] Remove staples left from the carpet installation with pliers. Look over all areas of the stairs to find holes and deep scratches.\",\"endings\":\"['[title] Get rid of any floating debris and knock out any plumbing fixtures, doors or fittings. [step] Also be sure to remove any railings, cabinets, or sections attached to the basement above ground.', \\\"[title] Pound on the stripped carpet with a hammer. [step] In most cases, you'll encounter gentle taps caused by hammering along the floor.\\\", '[title] Use putty or wood filler and a putty knife to fill in holes. [step] If you have a cement staircase, you will want to fill holes with epoxy.', 'Remove any tread tiles or other fixtures that are covered. Keep the stairs cool so that water and moisture can flow freely in the stairs and help them to dry.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}