{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"53acfdc9-e950-5ca3-8c9e-0f310dca9107","task_key":"default--validation--17178","task_revision_id":"1","upstream_id":"17178","short_description":"HellaSwag validation 17178","config":"default","split":"validation","body":"{\"activity_label\":\"Holidays and Traditions\",\"ctx\":\"[header] How to make a button christmas tree [title] Paint a styrofoam cone with green acrylic paint. [step] Use a paintbrush or a foam brush to apply a coat of green acrylic paint. Allow the paint to dry, then apply a second coat, if needed.\",\"endings\":\"['[title] Secure a flower garland over the tree. [step] Trace the green acrylic paint onto a piece of cardboard, then cut the stalks off.', \\\"You can use painter's tape to tape the red and green felt tracks on the base of the cone to ensure it holds up. [substeps] It is alright to use black acrylic paint, but it's generally best to use a less intense color.\\\", 'You do not need to paint the bottom of the cone. [substeps] The height of the cone is up to you.', \\\"[substeps] For every third coat, it's important to paint the outer edge of a styrofoam cone. Use a single color, and let dry to an opaque green.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}