{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dba1a3ec-a386-506c-aabf-04cd88686351","task_key":"default--validation--41384","task_revision_id":"1","upstream_id":"41384","short_description":"HellaSwag validation 41384","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to pop sorghum [title] Heat some oil in a frying pan. [step] Use a steel pot or a pan with steep sides. The exact amount of oil you need depends on the volume of sorghum you'll be popping.\",\"endings\":\"['[substeps] Make sure to fry for ten to fifteen minutes. Stir occasionally to ensure the oil cooks evenly.', \\\"Typically, you'll need two tablespoons of cooking oil for every ¼ cup (32 grams) sorghum kernels. Heat the oil on high until it appears shiny.\\\", 'For a medium-sized batch, use two or three packets of oil. For larger batches, use more or less oil depending on the size of your baked sorghum.', 'Stir constantly from time to time to speed up the cooking process, or you can give it a more thorough stirring. [title] Dunk the sorghum in oil.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}