{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a20ba6d0-1f01-5213-8879-7152bf02514c","task_key":"default--validation--2923","task_revision_id":"1","upstream_id":"2923","short_description":"HellaSwag validation 2923","config":"default","split":"validation","body":"{\"activity_label\":\"Playing violin\",\"ctx\":\"The woman talks throughout the entire video as she's demonstrating her movements or pointing towards her finger motions. eventually the woman\",\"endings\":\"['stops talking and ends with a smile and the words \\\" howcast original \\\" appear on the white screen.', 'shows a square of paper and explains the process before showing the process once more.', 'gets off the mat and drops to the first mat and leans into the air before hitting it again with her hand.', 'is seen in front of a billiards table as the girl is sitting down in front of it sporting a black eye, an ear pan and a face mask.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}