{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1711d0a1-2b4f-5917-9f56-770e2bb5d64e","task_key":"default--validation--17646--variant-3ef97fa11a1f","task_revision_id":"1","upstream_id":"17646--variant-3ef97fa11a1f","short_description":"HellaSwag validation 17646","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to draw blueprints for a house [title] Have a decent idea of your desired outcome. [step] Before starting on paper, it's going to help to get a basic idea of what the house is going to look like. It is important to know how many rooms are needed and how many floors are going to be put in the house.\",\"endings\":\"['For example, if the house is just going to be in the basement, then a measurement should be just to ask for time. [title] Make a measurement using a tape measure and tape recorder.', '[substeps] Make sure you have put some thought into the facility and that the party rooms are lined up and no more than 6 volunteers are needed. Figure out both the average fee for a living room, a budget or any number of amenities.', \\\"[substeps] Rooms can range from complex to cramped. It's best to get a general idea and plan to make your house totally bigger and more perpendicular.\\\", '[title] For the perimeter, use a pencil to draw the outside walls of the house. [step] Try to take up the whole grid paper to get a larger scale.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}