{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ef7e91f8-5ff8-5adb-bab2-7e9cbfb46f6c","task_key":"default--validation--38908","task_revision_id":"1","upstream_id":"38908","short_description":"HellaSwag validation 38908","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"A little girl is in the back seat of a vehicle while eating an ice cream cone. Ice cream gets on her nose as she's licking the cone. as she\",\"endings\":\"['bites the ice cream she gets in deep trouble with spreading cream to her face causing her baby sister to knee her.', 'continues eating, she takes off the lid of the ice cream cone that was in her mouth.', 'continues to eat her ice cream, she gets some on her arm and licks it off.', 'continues eating, her lip is cracked and bleeding, as well as the ice cream cone off of her chin.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}