{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e549862d-1783-5163-9665-d62ef7b4e4db","task_key":"default--validation--15054","task_revision_id":"1","upstream_id":"15054","short_description":"HellaSwag validation 15054","config":"default","split":"validation","body":"{\"activity_label\":\"Using the pommel horse\",\"ctx\":\"A gymnast is patiently waiting to start his performance. He jumps on to the board and starts spinning really fast before he jumps off and walks around. he\",\"endings\":\"['take a breath and the audience claps for him before he jumps back on it and starts spinning again.', 'goes back and forth for some time and then takes off again.', 'goes to different places and comes back and does the same thing for three more times.', 'spins around to one side to catch his balance as he takes his time to prepare.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}