{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a6e91baa-96aa-5103-bef4-172867c43ec2","task_key":"default--validation--39607","task_revision_id":"1","upstream_id":"39607","short_description":"HellaSwag validation 39607","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make biscuits from scratch [title] Dissolve the yeast in water. [step] Combine the warm water and yeast in a large bowl, stirring gently only to dissolve the yeast. [substeps] The water should be between 110 and 115 degrees fahrenheit (43 and 46 degrees celsius) in order to activate the yeast properly.\",\"endings\":\"['[title] Add the butter and stir until fully dissolved. [step] Begin adding butter to the bowl of dough, stirring until the butter is fully incorporated into the dough.', 'Use a food thermometer to check the temperature of the water before you add the yeast. [title] Add the sugar, butter, milk, egg, salt, and whole wheat flour.', 'You can also consider adding salt or baking soda to the warm water solution. You might even consider adding the mixture over the course of an hour as it improves.', 'Working with boiling water directly will be most effective because the heat engages the dough and allows it to rise. Fill a large bowl with the warm water.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}