{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa6828b8-ac09-56a7-8674-6cfe78afaa40","task_key":"default--validation--8000","task_revision_id":"1","upstream_id":"8000","short_description":"HellaSwag validation 8000","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to fix a leaky basement [title] Inspect and identify source of leak (i.e. [step] Crack in the wall, leaking window, etc. ) look carefully at organic surfaces such as wood, drywall, and carpets in a finished basement. [title] Dig down 6 feet (1.8 m) to the foundation all around outside of the house.\",\"endings\":\"[\\\"[step] Even if there aren't any leaks, the area will likely be filled with dirt and rotted manure to make it look less attractive to the landowner. The more information you can provide, the more effective this project will be.\\\", 'The width of the trench should be enough for doing some other steps on the bottom. [step] You need enough space for patching the foundation and installing draining pipe.', '[step] Wash up area thoroughly first, and it will take care of the matter for your second attempt. Once you get to the foundation, gently ease the ladder behind the roots and gently move it outwards to check the floor level.', '[step] With no room for the depth of the crack, dig into the foundation even more. Then your door should have a hole roughly 24 inches (61 cm) deep.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}