{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"844eb278-6133-5dff-b345-c4b59061fa76","task_key":"default--validation--21706","task_revision_id":"1","upstream_id":"21706","short_description":"HellaSwag validation 21706","config":"default","split":"validation","body":"{\"activity_label\":\"Cutting the grass\",\"ctx\":\"A man in a yard starts his lawn mower. the man\",\"endings\":\"['starts mowing the lawn with a red paint sprayer.', 'starts mowing the lawn in a racer and someone else starts mowing the lawn behind him.', 'mows his lawn with headphones on his ears as his dog walks around the yard.', 'slowly begins mowing the yard using the remote control.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}