{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2af718eb-23c1-579d-bef4-7651818198da","task_key":"default--validation--33443","task_revision_id":"1","upstream_id":"33443","short_description":"HellaSwag validation 33443","config":"default","split":"validation","body":"{\"activity_label\":\"Playing pool\",\"ctx\":\"The woman makes the shot and they celebrate with a high five. The man is telling the woman how to make another shot. she\",\"endings\":\"['attempts the shot and makes it and they high five again.', 'shows two different turns to make more shots.', 'steps down and not makes the shot.', 'makes the shot again and puts more challenge into her shot.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}