{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3db4bcd9-6f54-53ee-890f-16805935a9ce","task_key":"default--validation--15226","task_revision_id":"1","upstream_id":"15226","short_description":"HellaSwag validation 15226","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to care for yorkshire terriers [title] Start training your yorkie early. [step] Begin training your yorkie as soon as you bring him home. Devote plenty of time to teaching your dog the house rules (like where you want him to toilet) and any other social rules you want him to follow (like no barking at people).\",\"endings\":\"[\\\"If you can't give plenty of time to training, consider whether you should adopt a yorkie. [substeps] Many people return their dogs to animal shelters because they don't realize the time, cost, and effort needed to train.\\\", 'This training will ensure your yorkie gets used to his new home right away, rather than just trying to stick down a household schedule. [substeps] The dog should be walked around the home, shaved, and treated regularly.', 'Make sure you teach your yorkie basic commands like \\\" toilet, \\\" \\\" lie down, \\\" and \\\" stay \\\" at night. Start by teaching him the home rule, and reinforce it when it becomes harder.', \\\"Patience is a key element in training your yorkie, and it's the first step towards ensuring that your yorkie is safe and happy in the house. [title] Train your yorkie every five months.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}