{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"283a9464-bf75-5f9e-965a-9b3daa9e9350","task_key":"default--validation--26681","task_revision_id":"1","upstream_id":"26681","short_description":"HellaSwag validation 26681","config":"default","split":"validation","body":"{\"activity_label\":\"Beer pong\",\"ctx\":\"The second man magically has two pong balls in his hands then magically adds 10 cups to the table for beer pong. The first guy pours beer and the guys play beer pong. the second guy\",\"endings\":\"['takes the cup of beer with both hands, then he turns over the cups, drops it in the table for beer pong, then shoots balls into them.', 'makes shot after shot and the first guy drinks beers.', 'gives a boy about 15 cups of beer before throwing.', 'sits on the floor and he has one of the guys throw balls to one another.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}