{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6d5cffb-08b1-565a-8d92-20fd6273aae4","task_key":"default--validation--27280","task_revision_id":"1","upstream_id":"27280","short_description":"HellaSwag validation 27280","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"He lifts a sharpening stone from the yellow green bowl filled with water and talk about if for a while. he\",\"endings\":\"['sets the cream into the bowl and pours the cream into two glasses and adds sugar.', 'then puts mustard, some other vegetables in the yellow bowl and rubs it around the bottom to sharpen the knife.', 'proceeds to say a few other things about sharpening knives while voleyball balls are launched into the air.', 'lifts the knife and starts sharpening the knife with the stone and the talking continues.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}