{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8530caa3-e24f-5744-ae6e-ddf6038ce2eb","task_key":"default--validation--38512","task_revision_id":"1","upstream_id":"38512","short_description":"HellaSwag validation 38512","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"Blocks of butter is shown, then a person in white apron put block of white solid mixtures in a giant mixer, then a lady in white shirt and black cap is shown holding a bottle with red lid on it. while the mixture\",\"endings\":\"['is in the mixer, the man put in a lemon clip a spoon of the mixture, pour water out, with hot water, put in jam, fry and butter.', 'is mixing, the lady pour water from a metal container, then a brown liquid, then the mixture turned into yellow, then the lady had the mixture in a giant white plastic container.', 'is being blended, we are shown the tools to make the mixer.', 'puts in the mixer you measure it and serve it with cream, lemon leaves and mustard to go with the blend as well as cut up some sugar, a lemon and chocolate and mix it in olive and lemon juice.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}