{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"199e9741-ac73-52ac-9e72-506d08e5493e","task_key":"default--validation--11403","task_revision_id":"1","upstream_id":"11403","short_description":"HellaSwag validation 11403","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to flip a pancake [title] Wait for the edges to brown and the top to bubble. [step] When the edges of the pancake begin to crisp up and the top begins to bubble, it's an indication it's ready to be flipped. Pay close attention to your pancake to see when this happens.\",\"endings\":\"['[substeps] Check the bottom of the pancake before you flip it. If the fillings start to cook and the top is very close to popping up, it means the pancake is done cooking.', \\\"[substeps] If you're still not noticing the color of the spots under the roller. You should also try checking when the center line spreads outward towards the back of the pancake.\\\", '[title] Prepare the pancake with a knife. [step] When you see the edges brown, the pancake is ready to be flipped.', \\\"[substeps] If you keep burning your pancake, lower the heat when you try to cook the next one. If the center of your pancake is raw, then it means you're cooking it on too high of a heat.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}