{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"075b2745-3d16-57bf-9ae7-6a2085e6abe2","task_key":"default--validation--45489","task_revision_id":"1","upstream_id":"45489","short_description":"HellaSwag validation 45489","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to use a pogo stick [title] Make sure that the pogo-stick is in good condition. [step] Check the pogo stick for loose parts before you get on. Make sure that everything is secured and fastened properly.\",\"endings\":\"[\\\"[title] Take hold of the pogo stick's handle with one hand. [step] Place your other hand on the side of the pogo stick where the handle's prongs are.\\\", \\\"You don't want an important piece to fall off while you are jumping. [title] Make sure you are within the proper weight limit.\\\", \\\"[substeps] Slip the pogo stick around your finger and tap on any loose parts. If it's stuck, push and try again.\\\", 'They are also easy to disassemble if you notice any strange looking parts. [title] At the check-in point, carefully fill in the details on the pogo stick.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}