{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6fa07ec7-4633-525f-8586-08f48ee164c5","task_key":"default--validation--35913","task_revision_id":"1","upstream_id":"35913","short_description":"HellaSwag validation 35913","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"He then takes that mixture and adds it to a larger batch of cream which he freezes until it hardens. When the ice cream is done he tastes it and describes how good the flavor is. when the video\",\"endings\":\"['is finally over another text appears that present money is being credited to the man who made the smoothie, and why.', 'ends he says again while putting the ice cream in the sink.', 'is finished the man is shown smiling at his camera which has been in the lower left part of the screen with a screen address next to it and a website via the website pet protection services.', 'is over the closing credits and graphics are shown on the screen.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}