{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c08d0266-da0a-5cf2-b984-b7f4f3c0042f","task_key":"default--validation--31372","task_revision_id":"1","upstream_id":"31372","short_description":"HellaSwag validation 31372","config":"default","split":"validation","body":"{\"activity_label\":\"Ice fishing\",\"ctx\":\"A man is standing on ice with his pole standing straight up beside him. the man\",\"endings\":\"['got the pole and threw it and started skating.', 'grabs his fishing pole and starts fishing in the hole in the ice.', 'is then sitting and a blue bucket of water is shown, followed by some white clothes and the man preparing to pole vault.', 'looked off to the left and suddenly started spinning his pole making it go up higher into the air.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}