{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"84c8020e-585e-5ea1-8f3d-cc7c16ff315a","task_key":"default--validation--3547","task_revision_id":"1","upstream_id":"3547","short_description":"HellaSwag validation 3547","config":"default","split":"validation","body":"{\"activity_label\":\"Running a marathon\",\"ctx\":\"A large group of people are seen running around a disney resort while waving to the camera and several shots of the kingdom being shown. many characters\",\"endings\":\"['are shown waving to the runners as well as more shots of people running past characters and going through the finish line and celebrating.', 'are then seen playing with an animal along with cheering in the end.', 'then play a game of rock paper scissors over the various obstacles and fails and ends with them all putting on their parachute and zipping up their tents.', 'are seen playing a game on a beach with one another and back again at one point axl aff wins up against one another.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}