{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"59bd9f55-c458-5c14-b707-e20cad088e7c","task_key":"default--validation--41509","task_revision_id":"1","upstream_id":"41509","short_description":"HellaSwag validation 41509","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"The man grabs the fish from the ice and puts it right back into the hole he just pulled it out from while it's still attached to the hook and his wire. the man\",\"endings\":\"['then holds the fish and pushes the fish away from the camera.', 'then begins to pull on the fishing wire, pulls his pole up and and down a few times, and then adjusts the camera.', 'continues to fish around and takes off the camera to show himself off.', 'seems to have a problem with different fish and spins the haft of the reel between his fingers as he tries to find the right hole to pull him out.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}