{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0968c791-af7c-5b35-99b4-a6b8d2e677d9","task_key":"default--validation--23622","task_revision_id":"1","upstream_id":"23622","short_description":"HellaSwag validation 23622","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to put a lead rope on a horse [title] Approach the horse from the front. [step] Always approach the horse from the front and slightly to one side, where the horse can see you coming. Stand next to the horse's head, facing the same direction as the animal.\",\"endings\":\"[\\\"If you approach from the side (from its rear), the horse will not know where you are coming from. [title] Step through the rope to the horse's right.\\\", \\\"If it sees that you are approaching from the front, it will then ask for the lead rope. [title] Reach into the horse's mouth and remove the lead rope (s).\\\", 'Your hands should be about shoulder-width apart, but you can space out them a bit in order to prevent placing a lead rope on the horse. [substeps] If the horse cannot see you, it will be less likely to treat you as a threat.', '[substeps] If the horse turns away, click your tongue until the horse faces you. You may convince it to turn around with a food reward instead, if the horse is trained and gentle.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}