{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a4b69f5d-472f-57c8-ac91-dce886b47482","task_key":"default--validation--41066","task_revision_id":"1","upstream_id":"41066","short_description":"HellaSwag validation 41066","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to make a gin martini [title] Gather your tools and ingredients. [step] To mix this drink you will need a chilled cocktail shaker, a chilled cocktail glass, a cocktail spoon, a cocktail strainer, a jigger, chilled gin, chilled vermouth, plenty of ice, and an olive. [substeps] Measure out 4 small jiggers of gin.\",\"endings\":\"['They should be about the same size as straws, plus an extra extra if needed. Optionally add 1 teaspoon of salt in.', 'Measure out 1 small jigger of vermouth. [title] Place your ingredients in the shaker.', 'Place the jiggers into a cocktail shaker and screw the lid on. [title] Measure and pour 2 ounces (120 ml) of gin in the shaker.', 'You should use one shaker for you gin, and one for your olive. Pour in all the ingredients and stir everything together well with a cocktail spoon.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}