{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"51066593-4a9a-5b4c-87e7-cef99ce675bd","task_key":"default--validation--33761","task_revision_id":"1","upstream_id":"33761","short_description":"HellaSwag validation 33761","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to make snowman earrings [title] Gather your supplies. [step] Some of the supplies you'll need, like the crystal encrusted beads, the headpins, and the earwires with mirror beads, probably aren't going to be lying around your home. Some craft stores will have everything you need, particularly any craft stores with large jewelry-making sections.\",\"endings\":\"['[substeps] The headpins will look like tiny pieces of silver and gold. The earshells will also be between the wires.', 'Everything comes in single tip sizes, so avoid getting [title] Thread the headpins onto your earrings. [step] Put one wire through one eye loop on the bottom and one in the other eye loop on the top.', \\\"Check your craft store's website to see what they have for you. [title] Unpin a 5-inch by 6-inch piece of metal from your craft store.\\\", \\\"If you're unable to find these items in store, you can order everything you need online. [title] Slide your silver bead on the headpin.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}