{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d4cc8881-9de1-5275-9f5b-d88a01fe1da1","task_key":"default--validation--26807","task_revision_id":"1","upstream_id":"26807","short_description":"HellaSwag validation 26807","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"An introduction comes onto the screen for a video about ice cream. Some people are standing by an ice cream truck talking about getting some ice cream. they\",\"endings\":\"['get some out of a carton and they eat it.', \\\"give instructions about how to use the ice cream and what you'll like it on.\\\", 'walk up to the truck window and get their ice cream.', 'go into a small ice cream dome, and a person starts to do tricks off of it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}