{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bb24da41-20a2-5edc-af6e-48c2e4afca8c","task_key":"default--validation--10587","task_revision_id":"1","upstream_id":"10587","short_description":"HellaSwag validation 10587","config":"default","split":"validation","body":"{\"activity_label\":\"Assembling bicycle\",\"ctx\":\"A person is seen sitting on a stool and piecing together several parts of a bike. the person\",\"endings\":\"['disassembles and rearranges the bike.', 'then grabs a spare tire and continues grinding the tiny parts together.', 'continues piecing the bike together and pans around the finished product in the end.', 'continues piecing out the parts while looking off into the distance.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}