{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e3650839-d718-5685-b4fe-4b59f627edc0","task_key":"default--validation--6232","task_revision_id":"1","upstream_id":"6232","short_description":"HellaSwag validation 6232","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"The man uses stacked pennies to get the angle of the knife to be sharpened and demonstrates how to sharpen kitchen knives using a stone. the man\",\"endings\":\"['uses an electric knife sharpener to sharpen the knife.', 'slides the knife on the stone while pressing the knife with his fingers.', 'then straps a loop of gold metal onto the handle of the knife and tightens it to cut the metal and leans forward to prepare the bend.', 'sets down the knife and continues talking while demonstrating sharpening the knife.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}