{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"84a29ba3-4e72-5e59-889f-177efda5fcb1","task_key":"default--validation--32586","task_revision_id":"1","upstream_id":"32586","short_description":"HellaSwag validation 32586","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to belay [title] Make a loop in one end of the climbing rope. [step] Most professional climbers recommend tying in the climber with a figure 8 follow through knot. Start by making a loop, or bight, about three feet (one meter) from the end of the rope.\",\"endings\":\"['[title] Bend the tail around the rope below the loop. [step] While holding the loop you have just made, pass the tail of the rope around the front of the main strand a little bit below the bottom of the loop.', '[substeps] Check that there is enough rope left in the loop to tie off the climber. If there is not enough rope left, leave about six inches between the knot and the rope.', '[substeps] Remove your foot from the loop at this point to loosen the belay attachment. Start fishing your foot out from the loop.', 'You can make a wide loop in this loop, or create a narrower loop by tying a cord or a lever in the loop. [substeps] Make the knot about three inches (five centimeters) from the end of the rope, so that you can continue jumping.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}