{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"02bcc5da-8900-57e7-a518-cf8eed788198","task_key":"default--validation--11198","task_revision_id":"1","upstream_id":"11198","short_description":"HellaSwag validation 11198","config":"default","split":"validation","body":"{\"activity_label\":\"Doing karate\",\"ctx\":\"A girl is standing in her home with her hands behind her back and her legs shoulder width apart. the girl\",\"endings\":\"['starts spinning around in circles on her shoulders.', \\\"then starts playing a clarinet, while holding the instrument, by moving her hand to the front not at all provoking the girl's advances.\\\", 'jumps on the bed and begins perform arts.', 'then firmly drops her hands to her side, brings her feet together and does a bow and continues to do a routine of various karate moves.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}