{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a78d13ed-1c53-51cc-8178-356d8414c3e4","task_key":"default--validation--45101","task_revision_id":"1","upstream_id":"45101","short_description":"HellaSwag validation 45101","config":"default","split":"validation","body":"{\"activity_label\":\"Calf roping\",\"ctx\":\"The gates open and the man flies out with is lasso chasing after a black and white calf. the man\",\"endings\":\"['throws the rope and catches the calf, gets off, ties the cow up and returns back on top of the horse.', 'stands on a rock of rock and ties the calf up with a rope.', 'is running and jumping over large rocks, then finds another calf and ties it up to the ground.', 'trails after the calf and compares the size of the calf and how big the calf is.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}