{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0508dc95-535a-59de-a587-c3034438bd8f","task_key":"default--validation--12216","task_revision_id":"1","upstream_id":"12216","short_description":"HellaSwag validation 12216","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"Three girls are seen standing behind a counter speaking to the camera with one spreading butter around a pan. Another mixes ingredients into a bowl with another girl's help and the girls lick the spoon used afterwards. they\",\"endings\":\"['change sides of the pan and make practice with each other in the sink.', 'then pass the clam to each other while holding the bowl.', 'continue mixing with one another while using their hands in the end.', 'pour the mixture into a pan, put it in the oven, then shortly take it out some time after.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}