{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"94dbf8ee-ecbf-5bd8-9174-7fe1fb9e3bab","task_key":"default--validation--2136","task_revision_id":"1","upstream_id":"2136","short_description":"HellaSwag validation 2136","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to care for snakes [title] Buy a terrarium that will suit the snake. [step] A terrarium is a glass enclosure for snakes. It looks like a fish tank with no water in it.\",\"endings\":\"['Make sure it is specifically made for holding snakes as they are very good escape artists and will likely escape without a secure lid. You may need to purchase a long or tall terrarium depending on the breed of snake you get.', 'Terrariums can be the best choice as they allow the snake access to the water (and substrate) of the tank, and provide a temporary head back to underwater. If your tank requires a lid, look for something that is about 8 or 10 feet (3.7 or 15.', '[substeps] Reptile aquariums, or reptile aquariums, are more natural in appearance, than aquatic aquariums. They are air-tight aquariums that are kept on neutral: a tank has to pass air currents at the top but water can drain out.', 'It has to be big enough for the snake to enter and exit the tank. Terrariums are portable and require less space, allowing for larger snakes to fit comfortably in.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}