{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"05085e88-c522-5555-b8ca-e619cfd1d8c9","task_key":"default--validation--49695","task_revision_id":"1","upstream_id":"49695","short_description":"HellaSwag validation 49695","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"The light shines bright on the right and scares the baby who begins to cry hard. The light fades and the baby calms down and looks all around the car. the baby\",\"endings\":\"['throws his right arm in the air.', 'laughs and it calms down and starts moving and feetfirst talking.', 'stands on the car nose to nose with the door handle and screams because her nose is broken.', 'stops crying then gets down and starts to play with the car.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}