{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"94655492-0176-5dee-bfa9-1a1aa3a26ada","task_key":"default--validation--31033","task_revision_id":"1","upstream_id":"31033","short_description":"HellaSwag validation 31033","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to hire a house cleaning service [title] Decide what you want cleaned. [step] You may only choose to have the service tackle particularly tough spots in your home, such as the kitchen and main bathroom. Some families will want the house cleaned from top to bottom.\",\"endings\":\"['Your home may require the service to redo, from fixing the floors to cleaning the living room and kitchen. [title] Decide how big or small you want the house to be.', 'However, some families may charge between $50 and $200 each time they hire the service. [substeps] Explore your budget and check the interest rates for your services to get a good estimate of the work required.', \\\"You'll also need to decide whether you expect services that may go beyond standard cleanings, such as wiping down the baseboards or washing the windows. [substeps] Many services will not clean windows, as they don't want to create streaks or damage.\\\", '[title] Ask friends, family, and colleagues about the pros and cons of hiring an in-home cleaner. [step] It is important that the process is covered by the laws set out in your state.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}