{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c40f83ae-ee0f-559c-bc8a-e40080d8b208","task_key":"default--validation--49518","task_revision_id":"1","upstream_id":"49518","short_description":"HellaSwag validation 49518","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"The man stops playing and begins to talk to the camera using hand gestures to point to the harmonica, at times. the man\",\"endings\":\"['stands and joins the replay, including the opened notes on the harmonica in white lettering and 4 red numbers at the bottom of the screen.', 'plays the harmonica while looking to the camera and periodically pausing to read the harmonica.', 'then begins playing again using his hands to indicate the type of note he is hitting.', 'continues playing and ends by not talking anymore.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}