{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2e5842fa-9ad6-5a35-912b-e474e8e6a251","task_key":"default--validation--3799","task_revision_id":"1","upstream_id":"3799","short_description":"HellaSwag validation 3799","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to wear a square scarf [title] Form a triangle. [step] Lay the square scarf flat out on the floor or on a table in front of you. [substeps] Fold it in half diagonally, so that it forms a triangle.\",\"endings\":\"['With the same folded right triangle, fold it in half again. It should be formed in the middle lengthwise.', \\\"Make sure that it has both creases facing up and away from you. Open the triangle it's open so that you can tell where to place the two triangular creases.\\\", \\\"Lay the triangle piece down on top of the square scarf. Have a helper help you make your square scarf's end.\\\", 'It does not have to be absolutely perfect. [title] Grasp the two longer ends of the scarf and pick it up.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}