{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c339de20-c72e-5938-8673-f5503393bcb2","task_key":"default--validation--10485","task_revision_id":"1","upstream_id":"10485","short_description":"HellaSwag validation 10485","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to develop arm strength for baseball [title] Use forty-five degree front lateral raises. [step] Stand in front of a mirror. Have a dumbbell in each hand (5 pounds each).\",\"endings\":\"['Bend your knees until your shins touch the floor. Make sure the dumbbells are level.', 'Your arms should be down at your side. Lift your arms together until they reach about shoulder height.', \\\"Bring the dumbbells overhead until you're about knee-width away from yourself (about 32 inches). Place one hand directly in front of your chest, and the other hand on your hips.\\\", 'Do two sets of ten or twelve revolutions with fifteen additional sets. [substeps] Stand with your feet shoulder width apart.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}