{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"071af022-761b-5d4f-a39d-567b46f52ebf","task_key":"default--validation--42105","task_revision_id":"1","upstream_id":"42105","short_description":"HellaSwag validation 42105","config":"default","split":"validation","body":"{\"activity_label\":\"Playing bagpipes\",\"ctx\":\"He begins to tap his foot along with the beat of his tune, he starts to tap his left foot in a back and forth rhythm with his right foot. he\",\"endings\":\"['places his left hand on top of his right foot and begins to rhythmically dance.', 'finishes, and smiles at the camera.', 'finished and lowers his hands and give the woman a big handshake.', 'gets closer to the camera and begins to read from the paper that one hand is enclosed in the newspaper and the other is gripping the orange tissue on the tape as he continues to play the tune with his head thrown back.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}