{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e8e047ee-0e6c-5e86-9b83-005ccefa9121","task_key":"default--validation--44374","task_revision_id":"1","upstream_id":"44374","short_description":"HellaSwag validation 44374","config":"default","split":"validation","body":"{\"activity_label\":\"Brushing teeth\",\"ctx\":\"The girl turns on the faucet to wet her toothbrush. The girl turns off the faucet, unscrews the toothpaste, and puts some toothpaste on her toothbrush. the girl\",\"endings\":\"['throws her toothbrush in the hand that is standing on the sink counter.', 'brushes her teeth as the run the toothbrush through her mouth.', 'sits on the bathroom floor, spits an object into her hand, and then wipes her mouth with a towel.', 'brushes her teeth and spit in the sink.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}