{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cbd55333-a703-5856-ad46-8947e64c4b52","task_key":"default--validation--28378","task_revision_id":"1","upstream_id":"28378","short_description":"HellaSwag validation 28378","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to make a sugar and vanilla scrub [title] Pour the 4 cups of pure white sugar into a bowl. [title] Pour in 1 cup of vegetable oil. [title] Mix until all the sugar is coated with oil.\",\"endings\":\"['[title] Pour 3 teaspoons of natural vanilla into the bowl, into the mixture. [title] Slowly pour in a tablespoon of aloe vera gel.', '[title] Add 3/4 cup of water to make a paste. [title] Spread the paste across the entirety of the scrub.', '[title] Add a dry teaspoon of oil to make the mixture thicken. [title] Roll the mixture in a thin layer so that it forms a smooth lump.', '[title] When it is glossy, add in 1 1/2 teaspoons of baking soda. [title] Pour in the rest of the sugar and continue mixing until all the ingredients have incorporated, leaving little paste left behind (see right underneath after adding the egg yolks).']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}