{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6fa4cd7e-cacb-5c53-b16e-510db81a8b95","task_key":"default--validation--4616--variant-92227b111c7b","task_revision_id":"1","upstream_id":"4616--variant-92227b111c7b","short_description":"HellaSwag validation 4616","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make an ice cream sundae [title] Choose your serving bowl. [step] While ice cream sundaes are delicious in any kind of container, choosing the proper bowl can certainly add to your experience. The traditional ice cream sundae vessel is the tulip sundae glass.\",\"endings\":\"['Save yourself the cost by choosing a small glass bowl for this experiment. [substeps] The center glass determines the temperature for your sundae, while the side glass determines the temperature for the ice cream.', 'However, you can also use boats, bowls, or cups. Use a bucket if you want-you can make your ice cream sundae as big as you want.', 'Any container is fine to use as a dessert sundae. Depending on the type of container you have, you will need to protect it at all times.', 'Whatever vessel you chose, round out the sundae at a time that is reasonable. [substeps] As little as possible, create a flower bouquet in the center of your sundae bowl when topping it with ice cream.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}