{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"55f0dd0b-16e7-5065-a0d6-18f29c2e5830","task_key":"default--validation--24144","task_revision_id":"1","upstream_id":"24144","short_description":"HellaSwag validation 24144","config":"default","split":"validation","body":"{\"activity_label\":\"Travel\",\"ctx\":\"[header] How to rent a vacation home rental next to disney world [title] Search for vacation home rentals which are privately owned and advertised by the owner, and whose owners (not managers) are residents and personally available to help with any unforeseen need both before and during your trip. [title] Call and talk directly to the owner. [step] Assess your comfort level with what the owner.\",\"endings\":\"['Do they take the time to treat you like family? Spend time conversing with the owner and ask a lot of questions. [title] Obtain a listing of emails so you can contact prior guests.', 'Ideally, it should be someone who is operating a. A restaurant may need some help paying for the costs of hiring staff or having a limited amount of space for the restaurant.', \\\"Hire someone with experience in the sort of thing you're looking for. [title] Find out where to park your car.\\\", \\\"If the owner lives close to you, consider reaching out to them rather than simply calling and asking. The owner may not be open to providing contact for an extended period if there's a problem.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}