{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c6840153-e3a2-5495-abcb-0439b5cf5921","task_key":"default--validation--32246","task_revision_id":"1","upstream_id":"32246","short_description":"HellaSwag validation 32246","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to buy a modular house [title] Choose a location. [step] When you buy a modular house, you need to own the land before the house is constructed. This means you will have to start shopping for an empty parcel of land to put your house on.\",\"endings\":\"['This will require some thinking up if you wish to build an entire rental home. This method may involve using an adapter that plugs into the telephone lines going to and from your property.', 'Buying your land is already a project, especially if you are planning to use it for the rest of your life. You need to make a decision as soon as possible to decide where to build your house.', '[substeps] Consider the other potential pitfalls of a modular house. If a modular house is over 16 acres or larger, a bigger house is going to need to be built on it.', \\\"It's a good idea to have at least a general idea of where you want to put your house before you start working with a builder. [substeps] Land is very expensive in some areas, so it's important to understand how much an average parcel will cost in the area you want to live in before you start planning your house.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}