{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b661555b-ebad-5ecb-9f58-186a743efe03","task_key":"default--validation--41420","task_revision_id":"1","upstream_id":"41420","short_description":"HellaSwag validation 41420","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make peach shortcake [title] Place the oven rack in the center and preheat the oven. [step] To ensure that the shortcake will bake evenly, check that your oven rack is in the center position. Next, set the temperature to 450 degrees fahrenheit (232 degrees celsius) to preheat the oven.\",\"endings\":\"['[substeps] Most ovens beep or flash an indicator light when the oven has reached the preheated temperature. Pay attention so you know when yours is ready.', \\\"[substeps] To test your oven's temperature, move the rack into the position you laid it on the oven rack when it is about halfway preheated. When you're done baking, let it unhover on top of the rack.\\\", 'Once it reaches its intended temperature, use a ladder to lower the rack into the preheated oven. Continue to check the temperature and make adjustments until your cake has set and set.', '[substeps] For healthy shortcake, you can bake the shortcake for two minutes at 450 degrees fahrenheit (232 degrees celsius). For a longer, healthier shortcake, wait a full half hour between baking times.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}