{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2b9142eb-0f80-563b-8005-887b8e6b6f45","task_key":"default--validation--25607","task_revision_id":"1","upstream_id":"25607","short_description":"HellaSwag validation 25607","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make gummy bears [title] Place 1/2 cup of cold water in a saucepan on the stove. [step] Don't start the stove yet. It is simply easier to mix the ingredients right in the pan since the gummy mixture will stick slightly if you try and transfer it.\",\"endings\":\"['[title] Add 20 grams, or 2 tablespoons, of unflavored gelatin to the water. [step] You can find this at most grocery stores in the baking aisle.', '[title] Add 1 tablespoon of milk. [step] Depending on your liking, you can use regular milk or whole milk.', \\\"You can use cold water from the tap. [substeps] If you don't have a thermometer , 1 cup can be substituted for the water, although it would be easier to create your gummy bears with a thermometer.\\\", '[substeps] Do not boil your gummy-giving mix right before you add it to the saucepan. The water in the pan is not only too hot, but it also has the potential to create steam that will quickly turn into gummy bears.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}