{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2c5161ed-259e-5605-b824-784682bc62b6","task_key":"default--validation--19165","task_revision_id":"1","upstream_id":"19165","short_description":"HellaSwag validation 19165","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to wear a beret [title] Tuck the brim under the beret. [step] Every beret has a brim that fits snug on your head and holds the beret in place. Tuck the brim up and under the excess fabric of the beret.\",\"endings\":\"[\\\"If the beret too loose is uncomfortable and won't pull over your face, then simply leave it that way. See the instructions included with your beret.\\\", 'Then puff out the fabric of the beret so it hides the edge of the beret. Tilt the beret to 1 side, with the front of the beret pulled down to your eyebrow.', 'Tug at the brim to make sure everything is centered. Adjust to fit the beret snugly and block a few inches of vertical space between the brim and the rest of the frame.', 'Make sure you tie it in the center of the beret so that the beret is not dangling down. [title] Keep the beret well in place.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}