{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b3369d15-c3f2-52bc-a542-ece679d8c5f7","task_key":"default--validation--12434","task_revision_id":"1","upstream_id":"12434","short_description":"HellaSwag validation 12434","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to bake cupcakes without a cupcake pan [title] Create a stable structure using cupcake liners. [step] If you're using paper liners, you'll need to use at least 2 or 3 stacked on top of one another so that they're able to stand on their own. Foil liners are the best choice as they're already very strong.\",\"endings\":\"['[title] Place cupcake liners in warm oven. [step] Start preheating an oven to 375 degrees fahrenheit (180 degrees celsius).', '[substeps] A 9 x 10 inch cupcake pan is ideal for baking cupcakes that will remain upright for long periods of time. The liners may be made out of aluminum, glass, parchment, or even tin.', \\\"Make sure the liners are cut to approximately the same dimensions. [substeps] You'll want to use cut liners on bigger cupcake liners so that they all stand on the same vertical surfaces.\\\", \\\"[substeps] It's likely that you won't need to double or triple the foil liners. [title] Place the liners on a baking tray.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}