{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c059536d-e149-53e0-9c7f-744a3801ee38","task_key":"default--validation--27180","task_revision_id":"1","upstream_id":"27180","short_description":"HellaSwag validation 27180","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to throw a science fiction party [title] Make sci-fi invitations. [step] Make space-like invitations with star wars creatures, tin foil cut-outs, ufos, aliens, and anything else sci-fi you can think of. Send them out at least a week ahead of time so all your guests are sure to make it.\",\"endings\":\"['[substeps] If you send invites yourself, make sure your guests know there will be some sort of throw-in at the party. [title] Send out pokémon and ravaras at least a couple of weeks before the event.', '[substeps] You can even add star wars character pics if you like. Make sure when you make invitations that others follow them.', 'Try planning the order after the party (s) to have two already arrived by the time everyone leaves. Keep being and making and looking for inviting invitations that your guests can return to the party on an after.', 'Invitations should include your contact information for rsvp, time, date, and place of party, what guests should bring, and any other details. [substeps] You can choose a specific tv show to base invitations on.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}