{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4520442d-6ffa-50fd-86eb-5d164c40c1a5","task_key":"default--validation--36564","task_revision_id":"1","upstream_id":"36564","short_description":"HellaSwag validation 36564","config":"default","split":"validation","body":"{\"activity_label\":\"Making a sandwich\",\"ctx\":\"The mayonnaise is spread on the toasted bread and then the ham is placed and then cheese, then mayo is spread again, then the bacon is added and then the lettuce and tomatoes. the person\",\"endings\":\"['then sprays the lettuce and tomato along the sandwich a little measuring cup from the counter top.', 'whisks the lettuce and mayo into a glass bowl in the kitchen.', 'spread the ham and the tomato on the bread.', 'put sticks on the sandwich then cut into four, then he placed fries on the plate and the sandwiches.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}