{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"037ed389-5175-5412-97d7-2974b4635ac0","task_key":"default--validation--41644","task_revision_id":"1","upstream_id":"41644","short_description":"HellaSwag validation 41644","config":"default","split":"validation","body":"{\"activity_label\":\"Playing violin\",\"ctx\":\"The girl stops pulling the fiddle across the strings and walks closer to the camera to talk, and then resumes moving the fiddle across the strings. the girl\",\"endings\":\"['starts playing again and we see the boat sail past.', 'stops and looks back and grabs the mallet and start to continue playing while using a finger to perform.', 'then backs up, and turns to show her left side as she moves the fiddle across the violin strings, and then she moves to face forward again and walks closer to the camera and plays some more.', 'uses a third string and continues to speak up the bellybutton and shaking her hips from side to side as she plays and the audience is enjoying her performance.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}