{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1907fdd2-fdc8-5fe4-82a1-32c6ef51e25e","task_key":"default--validation--40666","task_revision_id":"1","upstream_id":"40666","short_description":"HellaSwag validation 40666","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to improve jarred spaghetti sauce [title] Start by sauteeing some diced onion. [substeps] Dice about half of a sweet onion per jar of spaghetti sauce. Place the onion in a pan large enough to hold the sauce you will complete.\",\"endings\":\"['Place two to three cups of diced onion in a saucepan over medium-high heat. [title] Cover the saucepan and simmer the sauce for 10 minutes.', 'On a large cooking tray lay a can of plain spaghetti sauce and two cups of water. Place the pot on the stove and turn the heat on high.', 'Cook this onion in oil, since you want to preserve soft spots. [title] Combine the diced onion and pan.', 'Put in olive oil with the diced onion in the pan. Use just enough to coat the onion (a couple tablespoons).']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}