{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3a9954ab-746c-549e-9402-a6a887a1bafc","task_key":"default--validation--36692","task_revision_id":"1","upstream_id":"36692","short_description":"HellaSwag validation 36692","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make tiger tails [title] Melt orange-colored candy melts in a microwave. [title] Dip a pretzel stick into the melted candy. [step] Place it on wax paper.\",\"endings\":\"['[title] Dip another pretzel stick in melted candy melts. [title] Sprinkle with melted orange candy melts.', '[title] Carefully brush the orange-colored candy melts on each pretzel stick. [title] Put the parchment paper on the coated pretzel sticks.', '[title] In a separate bowl, melt 1/2 cup of chocolate chips and 1/2 tablespoon of butter in the microwave. [title] Use a spoon to drizzle the chocolate over the now orange pretzel sticks to make stripes.', 'Roll the stick to find the exact length of your pretzel stick. You can fold it into the melted candy if you want as you see fit.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}