{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"71f50047-5280-5e3e-9283-43d2f473e558","task_key":"default--validation--23927","task_revision_id":"1","upstream_id":"23927","short_description":"HellaSwag validation 23927","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"A man in a denim shirt and black baseball cap plays the harmonica at the beach in front of a sandy shore, ocean water and a grass and flower covered knoll. the man\",\"endings\":\"['plays for some time while making sure that the harmonica is properly in putting on its proper place.', 'stops playing the harmonica and begins to talk while covered behind a scrim of foliage.', 'is standing at the top of a short wooden staircase, and, leaning against the railing the man begins to play the harmonica covering the entire instrument with both hands while playing.', 'continues to play the harmonica, then stops for a few seconds so he can read some of the harmonica lyrics.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}