{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65212fbe-b45b-54d6-9bc6-6685259d42d7","task_key":"default--validation--26012","task_revision_id":"1","upstream_id":"26012","short_description":"HellaSwag validation 26012","config":"default","split":"validation","body":"{\"activity_label\":\"Painting furniture\",\"ctx\":\"A woman is standing outside with some white furniture. She begins to point and explain the process to paint her furniture yourself like a diy project. after she\",\"endings\":\"['is done talking to the camera, she uses various tools to get soft materials while balancing and then wringing them out.', \\\"'s finished, she shows how you can hang the furniture, put the paint on it and spread it out in a decorative manner.\\\", \\\"'s done trying to paint, she begins putting all all of the pieces together and goes to get more supplies.\\\", 'shows the colors of the paint that she is planning to use then she grabs some grey paint and paint brush ans begins at the top of the white chair.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}