{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9f5bb985-596e-5f7e-9f9c-e297a8982c6a","task_key":"default--validation--30286","task_revision_id":"1","upstream_id":"30286","short_description":"HellaSwag validation 30286","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to make a snowman [title] Check for snow that's moist and packable. [step] If your snow is too fluffy or powdery, you will not be able to make a snowman. Go out to the snow and scoop some up between both of your hands.\",\"endings\":\"[\\\"Scoop about a tenth of it into your palm. If the snow looks really nice, that means you've fulfilled your promise.\\\", 'Check any size patches as well, as these tend to be less melter than dry, rubbery ones. [substeps] Remove hard ice from the bottom of your snow pile.', 'Scoop enough snow from the ground to seal it inside a plastic bag or container. [substeps] Cover your hands with tightly fitting plastic wrap.', \\\"Press it together, and if it forms into a ball, you'll be able to make a snowman. [substeps] If the snow falls apart, it's no good for making a snowman.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}