{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11132a81-1622-50c9-a575-865fce4b7e5d","task_key":"default--validation--49532","task_revision_id":"1","upstream_id":"49532","short_description":"HellaSwag validation 49532","config":"default","split":"validation","body":"{\"activity_label\":\"Getting a piercing\",\"ctx\":\"A woman and a doctor are seen talking to one another and leads into the person putting dots on her eyebrow. He shows her in the mirror and then pierces the girl's eyebrow and puts a gem through. he\",\"endings\":\"[\\\"then pierces the pinkie and then pierces the girl's right finger before sealing the jewelry on.\\\", \\\"continues to pierce the girl's eye and give her a shot.\\\", 'then cuts the top off and smiles to the camera.', 'wipes off the blood and shows the finished piercing to the girl in the mirror.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}