{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"588d9ef3-b750-5335-909d-e0272e5c3af9","task_key":"default--validation--38529","task_revision_id":"1","upstream_id":"38529","short_description":"HellaSwag validation 38529","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to get a cat used to snow [title] Build an outdoor cat shelter. [step] You can protect your cat from the elements and get her more used to the snow by building her an outdoor shelter for cold weather. You can use a storage bin that is just a bit bigger than your cat to create the shelter or build one out of wood.\",\"endings\":\"['You can then fill your shelter with layers of enough plastic so that your cat can sleep comfortably inside it. [substeps] Before building your outdoor cat shelter, you should get a good idea of the amount of snow your cat will be exposed to.', 'Alternatively, you can buy a cardboard box from a pet store. [substeps] Build the shelter on top of a cardboard box.', \\\"It's also a nice option to have a covered stand to protect your cat from the snow. [substeps] Keep insulation materials in mind so you can build adequate walls to provide a cool place for the cat to hibernate.\\\", 'Make sure the shelter is not too big, as you want your cat to be able to stay warm with her own body heat. [substeps] You should fill the shelter with stray or loosely stuff a pillowcase with packing peanuts or shredded newspaper and put it in the shelter.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}