{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"43aba9a9-6f98-5588-9594-fcb08e7ad30d","task_key":"default--validation--27594","task_revision_id":"1","upstream_id":"27594","short_description":"HellaSwag validation 27594","config":"default","split":"validation","body":"{\"activity_label\":\"Clipping cat claws\",\"ctx\":\"A woman is seen holding on a cat on a coach and holding up a pair of nail clippers. She begins cutting the cat's claws while the cat squirms around a bit. she\",\"endings\":\"[\\\"continues cutting the cat's claws while the cat lies motionless.\\\", 'continues cutting the nails and the cat begins barking.', 'continues cutting the claws while speaking to the camera.', 'continues speaking about the cat and shows off cat hair on her nails.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}