{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8ea01e23-95e3-5173-8e42-7f3b9e2d6021","task_key":"default--validation--46693","task_revision_id":"1","upstream_id":"46693","short_description":"HellaSwag validation 46693","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to eat out as a vegan [title] Check the menu for vegan options. [step] Before going out, check the restaurant's menu to see if they have any vegan or vegetarian options. If there is no vegan or vegetarian section, look for items listed as \\\" fresh \\\" or \\\" low calorie \\\" instead, as they often contain items with no meat or animal byproducts.\",\"endings\":\"['[title] Choose an organic option that lets you have all of the nutrients your body needs at a healthy level. [step] Realize that many diets are not as nutritious as it sounds.', \\\"Make sure you like the kind of food listed and what they are serving so you don't have to think of new foods to look for. [title] Visit the food at the center aisles.\\\", 'Some restaurants print animal-free options in a different color or mark them with a small symbol like a \\\" v \\\" or leaf. [substeps] For large restaurants and chains, check their website for current menu offerings.', '[substeps] Look at the prices if the menu says \\\" high protein \\\" or \\\" low fat \\\". Try this item if you will be dining on pizza and other low calorie foods.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}