{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"35d96e2a-1387-5dad-a7c6-d095245d6f4a","task_key":"default--validation--36031","task_revision_id":"1","upstream_id":"36031","short_description":"HellaSwag validation 36031","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to teach yourself a language using flashcards [title] Find existing flashcards. [step] Making and customizing your own is the most impactful way to learn, but you can also buy flashcards or find them online. Check retail stores or phone applications.\",\"endings\":\"[\\\"You'll find several excellent kinds of flashcards online that will help you learn a range of languages. [substeps] Ensure your palm, finger, thumb, and index finger are visible before inserting a flashcard.\\\", 'You can use programs to make your own simple flashcards that you use the same way as you would paper ones. This is at the very least an easy way to start before you tailor your own flashcards to your needs.', 'They are bound to be able to find flashcards for free or for a significantly lower price. [title] Make flashcards to test out a vocabulary word.', '[title] Choose a format and type of flashcard. [step] Either print a one-sentence message or format it by pressing ctrl + f.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}