{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65cbad0f-9be0-54d3-9ec7-339b4f03975b","task_key":"default--validation--48714","task_revision_id":"1","upstream_id":"48714","short_description":"HellaSwag validation 48714","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"On the late night show the guest is neil patrick harris, he is very excited to see a horse come out on stage and jumps up and down very happily. Once they sit back down they talk for a while and then neil starts playing the harmonica. he\",\"endings\":\"['stops and starts bouncing and gives a little wave then kisses the lady on the side of her face and gets up and goes back to playing the harmonica again.', 'keeps making random signs and faces at the camera as the show goes on.', 'then slows down and hits the drum a little as the song progresses to a faster clip.', 'seems to be pretty good at it because it continues to play for quite a while.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}