{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1ebefe04-94f1-57d9-83e5-c3c573e30f59","task_key":"default--validation--27593","task_revision_id":"1","upstream_id":"27593","short_description":"HellaSwag validation 27593","config":"default","split":"validation","body":"{\"activity_label\":\"Clipping cat claws\",\"ctx\":\"A woman is seen holding on a cat on a coach and holding up a pair of nail clippers. she\",\"endings\":\"['cuts the nails and holding onto the woman continuing to cut the cats nails.', 'clips the cat nails one by one as well as other cats and pets them.', \\\"begins cutting the cat's claws while the cat squirms around a bit.\\\", 'then begins painting the nails while moving away and speaking to the camera.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}