{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"014c2c19-e1ec-5993-92f7-801137104704","task_key":"default--validation--34069","task_revision_id":"1","upstream_id":"34069","short_description":"HellaSwag validation 34069","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make molasses dinner rolls [title] Combine the yeast and water. [step] In the bowl of an electric stand mixer, combine 4 teaspoons of active dry yeast with 1 cup of warm water. Let the yeast and warm water rest for approximately 10 minutes.\",\"endings\":\"[\\\"[substeps] The yeast and water should appear foamy after ten minutes; if it doesn't, the yeast is dead and you will need to start over with fresh yeast. [title] Add everything but the whole wheat flour.\\\", '[substeps] The active dry yeast has a light, chewy consistency. If you want a more of a smooth dough, you could add more active dry yeast.', '[substeps] Many people prefer warm alternatives to less-active alternatives, such as coconut or peanut butter, which make their baked goods more flavorful. [title] Combine the flour, yeast, and sugar.', \\\"[substeps] You do not need to keep the mixer running during this time. As long as the beaters and the mixer are equipped, the active dry yeast should reach the recipe's adequate volume.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}