{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7660b134-c78b-560d-8253-a03b146e465b","task_key":"default--validation--48124","task_revision_id":"1","upstream_id":"48124","short_description":"HellaSwag validation 48124","config":"default","split":"validation","body":"{\"activity_label\":\"Hand car wash\",\"ctx\":\"Two women talks inside a car washing shop where there are people waiting. The crew cleans cars by hand and in a machine. a worker\",\"endings\":\"['cleans the side doors of a cars while the interior is cleaned and the license plate directions read and the car joins the people in the car wash.', 'uses the machine to drive.', 'gives money to man who signs a paper.', 'work behind a car wash and is seen washing three car dealerships.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}