{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4db51e79-56df-53a7-9b34-7a810e8ebc89","task_key":"default--validation--11948","task_revision_id":"1","upstream_id":"11948","short_description":"HellaSwag validation 11948","config":"default","split":"validation","body":"{\"activity_label\":\"Cutting the grass\",\"ctx\":\"The person pushes the mower along the edge of the grass and throughout the rest of the grass. the man\",\"endings\":\"['stops and turns around to hand off the frisbee.', 'then grabs a weed wacker and cuts the weeds along the edge.', 'puts the machine on the grass and picks up some leaves to put on the lawn mower.', 'rolls down the lawn mowing path making turns and turns around.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}