{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aacab8b4-d127-56b8-b4b7-63a995590e21","task_key":"default--validation--39420--variant-1ccc1c9eb895","task_revision_id":"1","upstream_id":"39420--variant-1ccc1c9eb895","short_description":"HellaSwag validation 39420","config":"default","split":"validation","body":"{\"activity_label\":\"Work World\",\"ctx\":\"[header] How to perform an nsp (normal safety precaution) on the cadet l98a1 [title] Safety catch. [step] The safety catch is situated just above the trigger guard. If you can see the red ring on the left side of the rifle, then the safety is off.\",\"endings\":\"[\\\"If you cannot, do not attempt to release any pin, bolts, or other necessary trigger items before setting the safety catch. This will minimize the amount of precision you'll need to apply for the nsp.\\\", \\\"If you can't see the red ring, the rifle is safe. [title] Press the safety catch and make sure there is not a magazine fitted, remembering to use your left hand and to keep your right hand on the trigger guard at all times.\\\", 'If you can see the wires or pulleys, then the safety is off. [title] If you cannot see the safety catch button (red circle), press the bottom left button and then press the bottom right button (black circle).', 'If you cannot see the red ring, then the shoulder strap is on. [title] Place the back of your right arm over your left shoulder.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}