{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d4ffb39d-e6a8-57db-9e0c-443b8151042a","task_key":"default--validation--37509","task_revision_id":"1","upstream_id":"37509","short_description":"HellaSwag validation 37509","config":"default","split":"validation","body":"{\"activity_label\":\"Fixing bicycle\",\"ctx\":\"A person uses a bike chain pedals and a wheel to power a small generator. The camera zooms in on the machine parts while it's moving. the person\",\"endings\":\"['runs their hand along the machine parts to touch them.', 'twists the base on the bike.', 'stops turning the pedals as the wheel keeps spinning and the camera angle raises up above the machine and the video stops.', 'goes past several trees and speed bumps as they turn the machine on.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}