{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a1b81bdf-9af8-5511-83fb-f94eb016c468","task_key":"default--validation--34720","task_revision_id":"1","upstream_id":"34720","short_description":"HellaSwag validation 34720","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"He is standing in a kitchen in front of the counter that has a variety of raw vegetables displayed. He places a kitchen towel on the counter and then places the knife on top of it. he\",\"endings\":\"['uses another knife to paint his finger tips.', 'holds a red container and then places sticks of bread on the counter.', 'rubs the towel against the block then scrubs it along the knife.', 'then demonstrates how to use the rada sharpening tool to sharpen the kitchen knife by running iit back and forth in gentle and steady motion.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}