{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f878f7c6-5d57-580d-88c7-ee501b8f4c1b","task_key":"default--validation--12915","task_revision_id":"1","upstream_id":"12915","short_description":"HellaSwag validation 12915","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to build a fallout shelter [title] Gather the tools which include shovels and saws. [title] Assign different tasks to different individuals according to their strengths and weaknesses. [step] A good idea may be to have half of the group dig while the other half gathers the poles/logs.\",\"endings\":\"['[title] Choose an area that has no flammable material around it. [step] When a nuclear weapon detonates, it creates a thermal pulse which can set things 20 miles (32 km) away on fire.', 'Begin planning a meeting place to start your project about why each shovel fits in the group. [title] Include the team that will support the project, where private individuals will be addressing problems.', 'Once everyone arrives in the shelter, make sure they understand the situation when everyone is together. [title] Be up front about what you will be doing.', 'This means that when you reach the end of the first row, you take sides, engineer teams and engineers from each angle and place them together in a shape that looks like something interesting. [substeps] Proper timing will depend upon the construction.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}