{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"2f85db23-ea4a-56c3-b7ad-a3e7d9e68a61","task_key":"default--validation--1962","task_revision_id":"2","upstream_id":"1962","short_description":"HellaSwag validation 1962","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"There are people standing on a frozen body of water and are looking into a small hole that has a fishing wire in it and they watch it as it bobs up and down. the standing man\",\"endings\":\"['gets a brush that is attached to a pole and starts to move and gets a zip line out of that hole.', 'pulls the pole and wire up higher and the man kneeling on the ice pulls the fish out of the ice water and puts it onto the ice.', 'in red shirt starts to swing on the line and tries hard to get it to catch something and pulls on the wire several times and hangs from it like a seal.', 'is on the floating device and is swimming deep under the water, digging around with the pole, while spectators are standing nearby and he extends the fishing wire up to the hole he made before getting pulled under.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}