{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1be0f4b0-998c-5799-a450-d9322d0adad3","task_key":"default--validation--40884","task_revision_id":"1","upstream_id":"40884","short_description":"HellaSwag validation 40884","config":"default","split":"validation","body":"{\"activity_label\":\"Layup drill in basketball\",\"ctx\":\"A woman is shown speaking to the camera and point off into the distance. The camera pans out into another man standing on the court and two basketballs shown. the man and woman\",\"endings\":\"['side by side as the woman sits on a bench.', 'then shoot hoops with the balls and pass each other back and fourth.', 'begin playing hit and run and the woman does many tricks as well.', 'speak to one another while the woman begins throwing the ball onto the field.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}