{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fb9a3de4-ba64-5e0c-89d3-e8eeeece561f","task_key":"default--validation--36671","task_revision_id":"1","upstream_id":"36671","short_description":"HellaSwag validation 36671","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to hold an olympics closing ceremony party [title] Invite everyone you'd like to come. [step] It's best to keep a party like this informal, and perhaps even a drop-in occasion to let people come and go as they please. Send out email and text invitations or simply call people and ask them over.\",\"endings\":\"[\\\"[title] If you're having a sports event, put up posters. [step] You'll want your guests to know how much the activity means to you (ex.\\\", '[substeps] Encourage everyone to wear their favorite sporting gear, to add to the sporty theme of the party. If anyone is coming back from playing sport that day, ask them to stay in that mode! [title] Decorate the room where the party is held.', 'The fewer people a party is, the better. [substeps] A basic \\\" thank you for coming \\\" message and a polite address might work, but communicating with everyone over email or phone is not recommended.', '[title] Decide what you want to do with the event. [step] You may wish to create a warmup or turn it into an evening party.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}