{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0cc7a74c-59bb-5dfb-b8f6-942538a134ea","task_key":"default--validation--27033","task_revision_id":"1","upstream_id":"27033","short_description":"HellaSwag validation 27033","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to make vegetarian mushroom gravy [title] Saute the onion and mushrooms. [step] Heat ½ cup (120 ml) of extra-virgin olive oil in a large saucepan over medium-high heat. Once the oil shimmers, add ½ of a small onion that's finely chopped and 1 cup (4 ounces or 75 g) of finely chopped baby portobello mushrooms.\",\"endings\":\"['Pour the whole oil into the pan over medium heat until the shrimp and mushrooms turn dark. [substeps] Never use onion that is not minced, as it can be caramelized or seared.', 'Saute the onion and mushrooms for 8 to 10 minutes. The onion will soften and the mushrooms will release their liquid.', 'Allow the mixture to cook for 1 to 2 minutes before lowering the heat to medium and cooking for another 1 to 2 minutes. [substeps] In a smaller saucepan, you can easily mix with more olive oil.', \\\"Stir the contents of the pan well, so the onions and mushrooms are completely coated in the olive oil. [substeps] Don't worry if the mushrooms come out of the oil as you stir them.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}