{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0ad12748-f641-547f-8c2e-8f781facefec","task_key":"default--validation--27372","task_revision_id":"1","upstream_id":"27372","short_description":"HellaSwag validation 27372","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"A male chef appears in a kitchen standing behind an island with several knives on a block of wood with sharpening utensils on it. the man\",\"endings\":\"['then demonstrates the proper way to sharpen the knives, pierces one of the knives using the pointed knife.', 'then grabs the knife and starts gliding it over the block very gently and smoothly before hitting it over a silver steel stick.', 'then begins rubbing a steel plate into a wooden wooden horizontally lay surface and starts to sharpen the knife on it using several knives.', 'now pours water on the knife and then knives.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}