{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2839d2c5-c32d-58be-8571-569cfaa4b9ee","task_key":"default--validation--43833","task_revision_id":"1","upstream_id":"43833","short_description":"HellaSwag validation 43833","config":"default","split":"validation","body":"{\"activity_label\":\"Making a sandwich\",\"ctx\":\"A man, in black t-shirt, stands alone in a kitchen, talks to a camera in an animated way and prepares a peanut butter and jelly sandwich. the man\",\"endings\":\"['holds up a jar of peanut butter and a bottle of jelly after pulling out two pieces of bread while standing in front of a kitchen countertop.', 'slices a triangle shaped sandwich together with bread and put butter and peanut butter in a sandwich can.', 'demonstrates how to rub jelly on the sandwich with ketchup and stir fry in strips to fit the sandwich.', 'prepares peanut butter and jelly sandwiches consisting of sandwiches, cookies, salt and vinegar, ketchup and mustard.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}