{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"14ab8c5f-1d89-5f5f-87d4-87439eba0465","task_key":"default--validation--21117","task_revision_id":"1","upstream_id":"21117","short_description":"HellaSwag validation 21117","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to plan a romantic evening [title] Start early. [step] Get an early start by looking ahead and planning in advance. People get busy and schedules get filled up, so it is best to start planning at least a month ahead of time.\",\"endings\":\"['This is especially important if the romantic evening is for an anniversary or other special occasion. Begin considering activities and thinking about special moments that your partner will enjoy.', 'Try to begin planning about 15 to 20 days in advance and go straight to planning for the evening at least in advance. [substeps] To start, have a conversation with your family.', \\\"Generally people begin with a single date, even a month or two in advance. Don't expect to rush in to planning.\\\", \\\"[title] Decide what you like and what you'll do when you go out. [step] Don't plan plans so far in advance that you won't be invited to events.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}