{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"01be66bc-0b7a-52c8-9e32-ce1a1ca54c78","task_key":"default--validation--23654","task_revision_id":"1","upstream_id":"23654","short_description":"HellaSwag validation 23654","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make vegan mint chocolate cake [title] Start preheating the oven to 325 degrees f (160 c) as you coat your cooking pan with oil or cooking spray. [title] Place the beets in a saucepan and fill it with enough water to cover the beets. [title] Allow the water to come to a boil and cook them at a medium-high heat until they become soft when pierced with a fork.\",\"endings\":\"['[title] Add the fresh mint and let the beets simmer, stirring frequently, for a few minutes. [title] Cover with plastic wrap and allow the beets to simmer for up to five minutes.', '[title] Remove the beets from the pan and place in a bowl. [step] The sieve will remove any dirt, sweat or bacteria clinging to the beets.', '[title] Turn off the heat and place the beets in a large bowl. [step] Stir in the melted butter, chocolate syrup, and the vanilla extract.', '[title] Allow the beets to cool off the heat as you place a fine mesh sieve over a medium bowl. [title] Pour the water and beats into the sieve.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}