{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"56772284-ed47-57fa-9109-62be8fc1868c","task_key":"default--validation--27679","task_revision_id":"1","upstream_id":"27679","short_description":"HellaSwag validation 27679","config":"default","split":"validation","body":"{\"activity_label\":\"Shuffleboard\",\"ctx\":\"First the man wakes up with his eyes buck wide and he washes his face and gets dressed. Then he heads outside and plays a game of solo shuffle and he scores good points. he\",\"endings\":\"['then goes back to the state of his face but airs it so it looks like he just inhaled some turn it could be scotch it looks like he just spit it.', 'laughs his entire time and the other team keeps playing to keep score.', 'even pushes the ball using a strange angle and the ball still makes it to number 10.', 'completes the game, stands up and claps and the crowd cheers.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}