{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9027e0f5-7548-55d9-8edb-429a02d24404","task_key":"default--validation--43713","task_revision_id":"1","upstream_id":"43713","short_description":"HellaSwag validation 43713","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to seam carpet [title] Position the carpet well. [step] You should plan the seam so that it falls out in an area that gets very little foot traffic. For instance, placing it an area that will be underneath furniture is better than placing it in the center of your room.\",\"endings\":\"['[title] Pick up a piece of fabric to put on the seam. [step] The amount of fabric to use is up to your preference.', '[substeps] Hiding the seam also makes it less noticeable. Even a good seam may still be seen from time to time, so hiding it will make your room look more professional and neat.', '[substeps] Lay out a medium-sized section of your carpet going 45 °. For instance, if you put it on a dining table, it should be under a dining chair.', '[substeps] For instance, if your bedrooms are near the kitchen counter, you should place each staple in a corner of each corner. [title] Measure the area where you plan to either put a staple or seam.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}