{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"52f84f88-c531-54eb-939e-b6ad41f41174","task_key":"default--validation--48241","task_revision_id":"1","upstream_id":"48241","short_description":"HellaSwag validation 48241","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"A young man is seen sitting on a park bench playing a harmonica and tapping his foot. The man moves his foot up and down and pauses for a minutes before continuing to play. the man\",\"endings\":\"['sings a bit and still plays the harmonica.', 'moves his feet up and down and finishes his harmonica.', 'continues playing his harmonica and ends by wrapping his hands around his mouth and appearing to moan loudly.', \\\"continues playing all around the park acting as if he's a ghost and ends by walking away.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}