{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6ef28e65-3c60-55fa-80ac-03d1fa184a91","task_key":"default--validation--13509","task_revision_id":"1","upstream_id":"13509","short_description":"HellaSwag validation 13509","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A man speaking about how he will be sharpening knives today. He shows the tool that he will be using that is called chefs choice. he\",\"endings\":\"['shows the knife that he will be sharpening next so that he can see the increased sharpening point.', 'uses the knife forward, then side to side slicing tomato clips.', 'shows how to use it and how you can use it with either hand.', 'shows the sharpened knife that he will be sharpening.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}