{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"67dc9c6f-b44d-5f11-9342-bed1d1532f3a","task_key":"default--validation--17266","task_revision_id":"1","upstream_id":"17266","short_description":"HellaSwag validation 17266","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to maintain waterproof leather boots [title] Remove the boot laces. [step] Before you begin treating or waterproofing the leather, be sure to take out the laces. They'll only get in the way if you leave them in the boots.\",\"endings\":\"['Removing them will automatically decrease mud and dirt. To remove the laces, whip them around in your hand.', \\\"Plus, leaving the laces in will make it difficult to treat and waterproof the boots' leather tongue. [substeps] Set the laces aside in a safe place.\\\", '[substeps] If the laces are already dirty, you can skip this step. [title] Soak the laces in warm water.', 'Use a brush to untangle the laces and then clean them off. [substeps] If you are using a brush, make sure to use an excess or significantly more polish.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}