{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4924d4a1-5507-5588-ac46-73db1f426100","task_key":"default--validation--26446","task_revision_id":"1","upstream_id":"26446","short_description":"HellaSwag validation 26446","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to prevent infections in parakeets [title] Buy a big enough cage. [step] For one parakeet, your cage should be at least a foot and a half square. If you have two, you need at least a 28-inch-by-24-inch-by-36-inch cage.\",\"endings\":\"['If you have more than one parakeet, spend two dollars per cage for each starter bird. [title] Feed your parakeet the optimal food.', \\\"This will provide the active starter and sustain his flight on an unassisted basis. [substeps] A 23-inch cage is unlikely to provide enough room for one parakeet, but it will work if you've got two.\\\", 'Overcrowding can lead to certain diseases spreading more quickly, such as clostridial diseases. [title] Pick a well-ventilated room.', 'This cage should have room for a single parakeet at the same time. Parakeets move from cages with vertical bars to cages with single bars.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}