{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0d75ba0d-4d21-55ff-96e8-e064171a19a0","task_key":"default--validation--27527","task_revision_id":"1","upstream_id":"27527","short_description":"HellaSwag validation 27527","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to wear a ring sling [title] Pass the tail through the rings. [step] Begin by holding the sling up by the rings with one hand. The fabric should be smooth, and the tag should be facing up.\",\"endings\":\"['[title] Place the strap/elbow piece over your hand. [step] Transfer the strap, arm piece, and heel piece of the sling onto a work surface, and then position the prongs/screw posts that are located under the front post.', \\\"[substeps] Turn the handle of the sling counter-clockwise, which also secures the rings. If you're still holding the handles, ask other visitors to help you gather the wheels.\\\", 'Using your other hand, loop the tail through one of the rings, one until it comes out of the other. [substeps] Look at the tag for accuracy and understand how to tighten the ring unless otherwise told.', 'Take the tail and pass it through the rings. Then, pass the tail again over the top ring and through the bottom of the rings, as if you were threading a belt.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}