{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"20999553-6608-5bb1-8b7a-f345265f2d1b","task_key":"default--validation--17422","task_revision_id":"1","upstream_id":"17422","short_description":"HellaSwag validation 17422","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to identify a devon rex cat [title] Feel for a curly coat. [step] Devon rex cats are known for their unusual curly coats. Their fur grows in ripples and waves, often having a suede-like feel.\",\"endings\":\"['This coat is mostly wavy, usually in one or two layers, and has faded over time. Coat your devon rex cat with dog hair or cat hair to verify that it is a devon rex.', 'The fur grows out on top of their coats. To identify a devon rex, look for a short coat.', \\\"They are shorthaired cats, and their curls are short and dense, a bit like a poodle's. [substeps] A devon rex's coat can come in any number of colors.\\\", 'They also have thick distinctive coats that run the length of their bodies. [title] Identify a pom-pom with a curly coat.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}