{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e42c629e-16ec-55dd-b85a-73cc4752fb5b","task_key":"default--validation--39910","task_revision_id":"1","upstream_id":"39910","short_description":"HellaSwag validation 39910","config":"default","split":"validation","body":"{\"activity_label\":\"Clipping cat claws\",\"ctx\":\"A black and white cat sits on a table next to a girl wearing a black and white sweater. the girl\",\"endings\":\"['takes out a cell phone to get a cat claw and rubs the black and white cat with lube.', 'sits and pets the cat, as another woman is standing at the other end of the table petting her and smiles and claps like an old woman.', \\\"picks up a small black brush and sprays the animal's paw with a brand name.\\\", 'holds the cats paw in her left hand.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}