{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"16117aa2-f2d9-55ac-b94c-d67e4773bd89","task_key":"default--validation--29961","task_revision_id":"1","upstream_id":"29961","short_description":"HellaSwag validation 29961","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"The man then picks up a plate, flips it over and begins to run the blade of the knife across the protruding edges at the bottom of the plate. after\",\"endings\":\"[', the man really passes a plate up close to the castle wall where several pictures of colorful vegetables stand.', 'that he tries several other knives and eventually grabs a sharpener and begins sharpening it to the points that match the knife sharpener before surrning and laying the knife flat with one slice of the sharpener.', 'the knife is properly sharpened, the man does a hand stand and releases a cup of hot water into the sink.', 'doing this, the man then uses the knife to cut the tomato again, this time slicing neater and quicker, the man then smiles at the camera.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}