{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"90b68ddf-a0db-52de-b784-e6a1c661353d","task_key":"default--validation--38835","task_revision_id":"1","upstream_id":"38835","short_description":"HellaSwag validation 38835","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to homebrew malt extract beer [title] Wash your 16 to 20 quart cooking pot, spoon and measuring cup with soapy water. [step] Rinse it well with hot tap water. Make sure you get all the soap off.\",\"endings\":\"['[title] Bring about 2 gallons (7.6 l) of water to a boil. [title] Dump your can of hopped malt extract into the boiling water, stirring so as to not allow the thick malt to stick to the bottom of the kettle.', \\\"Remember you don't want to dirty any already oiled pot. [title] Place the beer into the filling tube, then gently fill the cup about halfway with water.\\\", '[title] Secure your pot lid on securely with clamp. [step] Turn on your medium sized microwave/stove in step three.', '[title] Measure your beer, then pour it into the boiling water. [step] Turn the stove burner up to medium, and wait for the water to boil, then pour the beer into the pot.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}