{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ac3ca25e-c82c-5222-b554-a5786c249697","task_key":"default--validation--19404","task_revision_id":"1","upstream_id":"19404","short_description":"HellaSwag validation 19404","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make sour cream coffee cake [title] Preheat the oven. [step] To ensure that the oven is hot enough to bake the coffee cake, it's important to preheat it. Set the temperature to 350 degrees fahrenheit (177 degrees celsius), and allow it to heat fully.\",\"endings\":\"['[substeps] You can use a thermometer to check the temperature of the oven. It should read at least 100f (75 c).', '[substeps] The temperature should be around 180ºf (18ºc) at this point. [title] Grease a 9 \\\" x 9 \\\" (23 x 10 cm) pan.', \\\"[substeps] Keep an eye on the oven so you know when it's fully preheated. Most models with flash an indicator light and/or beep to alert you.\\\", 'You can fill a cake pan with a little bit of cooking spray to help the cake cook faster. Once the cake is ready, remove it from the oven and set it aside.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}