{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"75430500-9a59-518a-8018-189fa6d9d2c6","task_key":"default--validation--35749","task_revision_id":"1","upstream_id":"35749","short_description":"HellaSwag validation 35749","config":"default","split":"validation","body":"{\"activity_label\":\"Brushing teeth\",\"ctx\":\"A small boy brushes his teeth at a bathroom sink. The boy stops brushing his teeth and starts manipulating the buttons on the electric toothbrush. the boy\",\"endings\":\"['rinses the toothbrush and his mouth with running tap water.', 'returns to brushing the buttons and stops to glower at the camera.', 'then picks up the razor and blows toothpaste down his chin.', 'then stops brushing and turns the savetop back off.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}