{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7f1cf2c9-aa43-5198-b4b0-3764d1b9da10","task_key":"default--validation--39801","task_revision_id":"1","upstream_id":"39801","short_description":"HellaSwag validation 39801","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to crack a \\\" master lock \\\" combination lock [title] Turn the dial at least three rotations clockwise first, just to make sure it is clear. [step] Turn the lock's dial clockwise to zero. [title] Apply pressure on the shackle, which is the curved handle at the top of the padlock.\",\"endings\":\"[\\\"[step] Turn the dial clockwise (cw) slowly as you press the shackle. If you can't turn the dial at all from the zero point, release the shackle and turn the dial cw just a number or 2 and then try it again.\\\", '[step] Squeeze the shackle in the same manner that you would shackle a small piece of hair; hold it in place while squeezing tightly. Continue squeezing until the shackle becomes slack enough to separate between two fingers.', '[step] Apply pressure enough that the shackle slides easily. A number of alternative lock combinations work to try the same number of turns.', '[step] Use enough pressure to lock the shackle to the right of the entry trigger. Turn the dial clockwise but keep your fingers spread flat to form a \\\" t.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}