{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ee0a3083-df9a-5582-8a2a-7cbb6907e995","task_key":"default--validation--48713","task_revision_id":"1","upstream_id":"48713","short_description":"HellaSwag validation 48713","config":"default","split":"validation","body":"{\"activity_label\":\"Playing harmonica\",\"ctx\":\"On the late night show the guest is neil patrick harris, he is very excited to see a horse come out on stage and jumps up and down very happily. once they\",\"endings\":\"['high five they hit a high flip and start to run away and everyone is watching them.', 'sit back down they talk for a while and then neil starts playing the harmonica.', \\\"are in place, he starts to very vigorously saddle up the horse and starts leading him around while he cuts his rope and curls it around and around the horse's body using some tricks.\\\", 'are on stage, they begin to perform a quick dance that leads into a man jumping off and the performance being shown again in slow motion.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}