{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e3331722-e915-5eca-9901-e3ccb7d1f89f","task_key":"default--validation--27519","task_revision_id":"1","upstream_id":"27519","short_description":"HellaSwag validation 27519","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"One of the girls begins by adding two cups of flour to a large green bowl. Then the little boy adds some raw eggs to the flour. the little girl in the middle\",\"endings\":\"['wets her hand and pans the bowl out for more, while the little boy continues to mops the floor.', 'dissipates, leaving just the two cups of flour.', 'continues to scoop the mixture up in the other cup.', 'stirs the mixture in circular motion.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}