{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3d7ab78d-c132-538b-8429-03c7ac3e8593","task_key":"default--validation--36528","task_revision_id":"1","upstream_id":"36528","short_description":"HellaSwag validation 36528","config":"default","split":"validation","body":"{\"activity_label\":\"Holidays and Traditions\",\"ctx\":\"[header] How to create a christmas wreath [title] Start by preparing the elements that will form the basis of the wreath around the oasis ring. [substeps] Cut small sprigs of blue pine, removing the bottom 2 cm/1 inch of needles. You can either trim the needles with scissors or remove them with your fingers.\",\"endings\":\"['Cut 30 sprigs of berried ivy. [title] Place the base wreath elements onto the oasis ring.', 'You can cut small shoots from similar trees if you wish, or pick up branches if necessary. Cut twice or three times at the base of the tree.', 'Remove the ribbon and tape if they are too long for the wreath. You can also use color-safe silk ribbon.', 'Another method is to cut strings of household lights. Stand the string approximately 6-10 inches from the end of the oasis ring.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}