{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"23084e19-2146-51b4-b2c4-f0db090ac6ef","task_key":"default--validation--21286","task_revision_id":"1","upstream_id":"21286","short_description":"HellaSwag validation 21286","config":"default","split":"validation","body":"{\"activity_label\":\"Swimming\",\"ctx\":\"The woman begins to emulate the man's actions, moving her arms and dipping her head under the water. the man\",\"endings\":\"['then appears out of nowhere and begins to hold the woman.', 'shows her how to swim outward by doing so himself.', 'reaches out and she jumps in, submerged at the waist.', 'licks his lips as the woman moves under the water in the same manner.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}