{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"841ec377-2159-5347-b444-2994a7e834fa","task_key":"default--validation--42276","task_revision_id":"1","upstream_id":"42276","short_description":"HellaSwag validation 42276","config":"default","split":"validation","body":"{\"activity_label\":\"Calf roping\",\"ctx\":\"The bull is released and a man on a horse chases after it. The man jumps off the horse and ties the bull up. the man\",\"endings\":\"['puts ointment on the bulls nose.', 'stands up and walks back to his horse.', 'ties up the horses legs as well.', 'then jumps back on the horse and ties its legs tightly.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}