{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"db82b5f3-2fbf-58b2-9a64-8fda52d6998b","task_key":"default--validation--39012","task_revision_id":"1","upstream_id":"39012","short_description":"HellaSwag validation 39012","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to buy shoes [title] Buy shoes from a brand known for quality and comfort. [step] Many well-established footwear brands are known for their reliable, functional footwear. Buy a pair of shoes from one of these brands to make sure that you are getting a high quality product.\",\"endings\":\"['Most of these companies have built their reputations on years of good customer service as well, so these shoes are likely to be covered by a generous warranty. [substeps] Find quality footwear brands by reading online reviews.', '[title] Talk to your shoe shop employees about new shoes. [step] This could change their shoes line or lower their sales pitch.', \\\"Check the store's label for details and tips on sneakers or other shoes. [substeps] Children should not wear their sneakers too frequently and normal public footwear should not be worn as shoes; however, kids who wear their sneakers for a variety of periods should generally wear sturdy shoes.\\\", \\\"Once you purchase shoes you would be prepared to spend most of your time living with them, although that doesn't mean that they will last forever. [substeps] If you're having a hard time finding good sneakers, try looking for good pairs of shoes at the store.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}