{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"45011b7b-7a51-5446-9cba-70d677ebe7a2","task_key":"default--validation--31548","task_revision_id":"1","upstream_id":"31548","short_description":"HellaSwag validation 31548","config":"default","split":"validation","body":"{\"activity_label\":\"Health\",\"ctx\":\"[header] How to stay safe when shoveling snow [title] Don't underestimate the task or overestimate your fitness. [step] Don't sell the exertion required to shovel snow short. If you would have trouble completing a good jog or weight-training session, you may have difficulty clearing your driveway and sidewalk.\",\"endings\":\"[\\\"It's important to make it clear to your neighbors that it's your work for them in case you drop on them again. [substeps] Don't say you'll be doing your job right after ice water starts to appear.\\\", 'Upper body workouts like clearing snow actually put more strain on the heart than activities like running or biking. [substeps] If you have any reason to doubt the adequacy of your fitness level, consult a physician before undertaking snow shoveling.', \\\"[substeps] If you've always told your friends that you don't need long walks in your neighborhood, this will save you money and time. If you really want to keep a snow shovel on you, build one until you have two vehicles available.\\\", '[title] Avoid using everyday tools, like electric tools or rakes. [step] Be aware that this will prevent you from being able to work it under the supervision of professionals.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}