{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"874cbdbd-6ee3-5809-b11b-7b4fa828af47","task_key":"default--validation--48113","task_revision_id":"1","upstream_id":"48113","short_description":"HellaSwag validation 48113","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make a white russian [title] Fill a glass with cubed ice. [step] Use a highball glass or something similar. Fill the glass to the brim with ice.\",\"endings\":\"['Place a coaster somewhere to rest the glass. [title] Place the sugar and butter in a mixing bowl.', \\\"Place a strainer under the glass so that the ice doesn't spill out. For a better touch, you can pour water over the ice before putting it on the ice.\\\", 'This will help diminish the bitterness of the ice and add some more sweetness to the drink. This glass is very fragile and will not break easily.', 'Use cubed ice over crushed ice, as cubes will not melt as quickly and prevent your drink from becoming watered down. [title] Add two ounces of your vodka.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}