{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"795bd755-4951-5687-a089-901ace359765","task_key":"default--validation--33441","task_revision_id":"1","upstream_id":"33441","short_description":"HellaSwag validation 33441","config":"default","split":"validation","body":"{\"activity_label\":\"Playing pool\",\"ctx\":\"The man and woman continue talking and the woman laughs at what the man is saying. The man is now teaching the woman how to shoot the pool ball while standing behind her. the woman\",\"endings\":\"['makes the shot and they celebrate with a high five.', 'holds the pool ball and the man hit the cue ball.', 'finally returns to talk to the man in the rear, but then she tries to shoot the pool ball from behind.', \\\"shoots and takes it's turn at the wall, then she gets frustrated and walks away.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}