{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c8ee7225-f657-5628-a537-46cd43fbc3c3","task_key":"default--validation--1689","task_revision_id":"1","upstream_id":"1689","short_description":"HellaSwag validation 1689","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to gate vault [title] Run up to a suitable obstacle (eg mesh fence. [step] ) the obstacle should be about head height any lower and it would be better to monkey vault it. Make sure the top is fairly thin so you can grip it well, or you may fall.\",\"endings\":\"['Climbing a fence not only causes injury, but it also scares people, so go running. [title] Hold onto the fence at all times.', '[title] Bring one arm up and then pull the other arm up into place. [step] Now pull your arm down and the other arm up while drawing it back.', '[title] Climb/wallrun/arm jump/to the top of the obstacle and pull yourself into a position where the top of the obstacle is about hip height and your arms are locked into position. [title] Lean over the object forward and to your dominant side.', '[title] Press the button down button on the remote remote to close the circuit. [title] Use the right side of the remote to lift up the gate.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}