{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f58b7be8-d173-5f0a-a3b9-fd22a3ebfbdb","task_key":"default--validation--12585","task_revision_id":"1","upstream_id":"12585","short_description":"HellaSwag validation 12585","config":"default","split":"validation","body":"{\"activity_label\":\"Playing violin\",\"ctx\":\"There are two young girls holding violins in their hands, standing with a music stand and a book with music notes behind them. the older girl\",\"endings\":\"['begins singing while playing the violins as the girl behind her starts to dance.', 'then attempts to play the violin, the older girl holds the violin too while the younger continue playing.', 'is helping and teaching her younger sister to properly hold the violin by changing her head position.', 'on the left reaction to a song by lisa anthony and the older girl on the left react to the song by annie jones.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}