{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6abc1594-f737-555a-9b41-aecce7ac4d70","task_key":"default--validation--30695","task_revision_id":"1","upstream_id":"30695","short_description":"HellaSwag validation 30695","config":"default","split":"validation","body":"{\"activity_label\":\"Playing pool\",\"ctx\":\"A close up of a pool table is shown followed by a person walking around the sides. The person then hits a pool ball continuously around the table. the man\",\"endings\":\"['then sinks the last ball and celebrates with a group of people.', 'continues to play and scoop out the balls while the camera follows his movements.', 'then positions himself and hits the cue ball and sits on the table to score the pool.', 'then walks away as well as a woman who hits the pool ball.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}