{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"20b6d7ee-acc1-5361-b177-d5916d305615","task_key":"default--validation--11563","task_revision_id":"1","upstream_id":"11563","short_description":"HellaSwag validation 11563","config":"default","split":"validation","body":"{\"activity_label\":\"Youth\",\"ctx\":\"[header] How to act like lyra [title] Change your attitude first. [step] Lyra is clever and canny. Learn as many things as you can and answer every question thrown at you.\",\"endings\":\"[\\\"Don't think that lyra is loud-mouthed, she is quite quiet but is a little louder around adults. Be cheeky, lie about things that you have done.\\\", 'Show your enthusiasm for everything, your experiences, and your ability to think critically and think about situations head on. Put a little work into first-think first-let what you thought last but changed many years ago make a point of it! When it comes to acting cool and confident, just pretend you have nothing to fear.', \\\"There is no need to be secretive, unless it's a horror story and you feel you cannot answer that question. She's most conscientious of her students and will never tell anyone anything.\\\", '[title] Always answer as many questions as possible. [step] Treat every question as if it were a really wrong answer and your answer will never have to be wrong in any part of your life.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}