{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8440efb5-ebfd-5539-9c51-338216f1fee0","task_key":"default--validation--27520","task_revision_id":"1","upstream_id":"27520","short_description":"HellaSwag validation 27520","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"Then the little boy adds some raw eggs to the flour. The little girl in the middle stirs the mixture in circular motion. then she\",\"endings\":\"['cuts up some mangos and puts them into a bowl with the egg.', 'goes to a mixing bowl to add more eggs in it many times.', 'adds some melted butter to the mixture and stirs it again.', 'scoops as around the pan and dumps around the dough.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}