{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d9a0227e-22a5-592b-8ee7-4395b7591c0d","task_key":"default--validation--24631","task_revision_id":"1","upstream_id":"24631","short_description":"HellaSwag validation 24631","config":"default","split":"validation","body":"{\"activity_label\":\"Spread mulch\",\"ctx\":\"A herb garden appears with a woman standing next to it in a large garden next to a wheelbarrow filled with mulch. the woman\",\"endings\":\"['moves the mulch across the ground in the wheelbarrow, falling backwards on attempts.', 'then begins to mulch close to the wheelbarrow with mulching tool in her hand and while waving her arms in the air.', 'begins to talk to the camera while gesturing to the flowerbed and the mulch, before eventually picking up a handful of the mulch.', 'takes some of the mulch away and starts bagging it in the wheelbarrow.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}