{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"50a1c4a7-c46b-52ec-96e6-2739d84408c4","task_key":"default--validation--36582","task_revision_id":"1","upstream_id":"36582","short_description":"HellaSwag validation 36582","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to make ankle bracelets [title] Gather your materials. [step] To make your ankle bracelet, you will need string. You can use one color, or an assortment.\",\"endings\":\"['[substeps] You will need to use twine and thread. Dye the string with any color you like.', \\\"You'll need three pieces of string for this type of ankle bracelet. You can find the string at craft stores.\\\", 'Make sure you gather either a weight or a type. [substeps] Ideally, some kind of thick string will work the best.', 'If you are using one color, check that the color goes through the jewelry stores. Yellow/orange gold, red, purple, or green beads are common choices.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}