{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"27959f30-c1a2-5e83-8e0e-fddd8f67e6d3","task_key":"default--validation--2758","task_revision_id":"1","upstream_id":"2758","short_description":"HellaSwag validation 2758","config":"default","split":"validation","body":"{\"activity_label\":\"Drinking beer\",\"ctx\":\"The man pretents to time himself and uses the screwdriver tool to pierce a hole in a full beer can that sprays some of its contents. The man holds up the can with the hole facing upwards. the man\",\"endings\":\"['wipes down the empty beer can.', 'then puts the can down, twists the cap off and pours the alcohol into a cup in front of him.', 'drinks the contents of the beer can from the hole after opening the top.', 'finishes by set the can down and shows off the spray can.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}