{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6324967c-6ace-5562-928f-e92b10827bca","task_key":"default--validation--7188","task_revision_id":"1","upstream_id":"7188","short_description":"HellaSwag validation 7188","config":"default","split":"validation","body":"{\"activity_label\":\"Playing guitarra\",\"ctx\":\"As he begins, a small rectangle appears in the upper right corner of the video and shows the strings as well as the note he is playing. the guy\",\"endings\":\"['takes a pause and then begins playing a song and the lyrics to the song appears at the bottom as well as the notes above them.', 'stops playing and begins drumming on the car while looking back at the camera.', 'ends the video and starts playing drums in the room until the video ends.', 'seems to have more than one set of notes after the move and is very worked up about his performance.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}