{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ada2af1f-1129-5919-adce-0c9ac1638283","task_key":"default--validation--45086","task_revision_id":"1","upstream_id":"45086","short_description":"HellaSwag validation 45086","config":"default","split":"validation","body":"{\"activity_label\":\"Playing rubik cube\",\"ctx\":\"A person's hands are seen attempting to solve a rubix cube while the end of a table is shown. the person\",\"endings\":\"['continues until they finish the cube and then shows his time on the screen.', 'grips the bar as well as the company logo and camera angles before finally solving the puzzle.', 'pushes a piece of paper down onto the table and mis-solves the rubix cube in a matter of seconds.', 'uses a tool to solve a rubix cube while the end of the table is shown.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}