{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"066ad5da-4564-540b-9331-1a2761dc1b65","task_key":"default--validation--32305","task_revision_id":"1","upstream_id":"32305","short_description":"HellaSwag validation 32305","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"He gently tries to pull up and then reel the fishing line out of the hole. He dips and lowers while taking a break. He tries to reel the line out again. he\",\"endings\":\"['looks up from his shallow hole rubbing his face.', 'pulls up, takes a few drags and starts reeling again.', 'removes his gloves and gets to his knees and back up as he tries to pull the line out of the hole repeatedly.', 'reels the fish out once more and pulls it out of the hole.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}