{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1c0cdf98-34e2-54a0-9fac-70229df8c2e0","task_key":"default--validation--18657","task_revision_id":"1","upstream_id":"18657","short_description":"HellaSwag validation 18657","config":"default","split":"validation","body":"{\"activity_label\":\"Hanging wallpaper\",\"ctx\":\"Various text is shown is steps across the video as well as a woman unrolling some paper and placing it along the wall. she\",\"endings\":\"['then lays out more paper and demonstrates how to do the exercise using various tools she seems to have.', 'places the paper along the wall and cuts the bottoms off.', 'begins to fold a paper out towards the wall while the camera turns back to her as well as the ceiling.', 'cuts the paper and rolls up the papers, making sure they clean off well.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}