{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"36879ff4-f4d0-52cc-951d-22cf85be18ed","task_key":"default--validation--8831","task_revision_id":"1","upstream_id":"8831","short_description":"HellaSwag validation 8831","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to choose the right insulation contractor [title] Licensed and insured. [step] The contractor you are hiring has a current license to perform work in the state and is insured to state regulations. [title] Experience.\",\"endings\":\"['[step] Architects, engineers, architects, architects, engineers, architects, engineers, engineers. The contractor should at least have a clear knowledge of the state and federal environmental regulation department.', '[step] The owner of the home can gather information about the owner by applying for the license. These like was needed to obtain property insurance, improve the value of the home, and determine whether the home is suitable for the home owner as a whole.', \\\"[step] That has been in business for a respectable number of years. There are contractors who have been around for over 35 years and you just can't beat experience.\\\", '[step] A contractor does not possess a lot of experience in a specific area. In addition, the contractor may not be certified by an accredited insurance company.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}