{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dad55c65-b94a-57d2-8693-ce0be70d0763","task_key":"default--validation--13649","task_revision_id":"1","upstream_id":"13649","short_description":"HellaSwag validation 13649","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to roast apples [title] Preheat the oven to 400 degrees fahrenheit. [title] Wash the apples. [step] Use a vegetable scrubber to scrub the skins, then pat the apples dry.\",\"endings\":\"['[title] Slice apples and apples into eighths. [step] Place each piece of apple between two bowls.', 'If you prefer peeled apples, peel your apples after washing them. Any type of apples will do, but fuji or granny smith apples are both great candidates for slicing and roasting.', 'Use a sharp, non-stick cooking knife to cut off any stubborn seeds. It is okay if some of the seeds come off of the apples.', 'Season with salt and pepper to taste, then place apples into the oven and bake for 15 to 20 minutes. When cool, transfer apples to a plate lined with aluminum foil and season with the finish of your choice.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}