{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8e327295-8cd3-5bc0-ae08-205fa853666e","task_key":"default--validation--40719","task_revision_id":"1","upstream_id":"40719","short_description":"HellaSwag validation 40719","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"Then he puts his fishing net in a hole in the lake and continues fishing. He catches another dark gray fish and tosses it in his box. he\",\"endings\":\"['catches a third fish from several different locations.', 'takes the static fish and tosses in his box of fish and watches the fish.', 'picks up another fish that is lying on the ground and throws it onto the edge of the lake.', 'continues fishing and catches more fish in that same hole.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}