{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2d70e695-6e63-51a2-bb94-a89991f9cfd7","task_key":"default--validation--44144","task_revision_id":"1","upstream_id":"44144","short_description":"HellaSwag validation 44144","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to touch up car paint [title] Buy a matching touch-up paint color. [substeps] Check the firewall (bulkhead) to find the paint code number. The firewall is the piece of sheet metal that separates the engine under the hood from the passengers inside the vehicle.\",\"endings\":\"['The number will also designate the name of the engine you will be touching up. [title] Call your locksmith to make a custom primer for your car.', \\\"If you don't have a matching touch-up paint code, observe the color coding in the firewall. This is usually indicated with a red and white square at the bottom of the panel.\\\", \\\"You will need to open your hood to find this number. Purchase a primer along with the touch-up paint unless the particular touch-up paint you've chosen states otherwise.\\\", \\\"The code is imprinted along the bottom of the firewall, beginning at the fully hot brights. [title] Look for the manufacturer's matching logo under the engine compartment (backside).\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}