{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5549fe0e-38e1-5720-b693-1689c4e8cf6a","task_key":"default--validation--18208","task_revision_id":"1","upstream_id":"18208","short_description":"HellaSwag validation 18208","config":"default","split":"validation","body":"{\"activity_label\":\"Drinking beer\",\"ctx\":\"A close up of a person is seen grabbing cans and poking holes into them. The man then begins drinking out of the beers all at once. he\",\"endings\":\"['wipes down the cans with his hand and shows off the, \\\" claps \\\" sign.', 'finally jumps up and continues drinking making different shots as well as waving to the camera.', 'continues on drinking beer and hangs out with another person hanging out.', 'drops the beers down at the end while people watch him.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}