{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8a319b09-7db4-55ac-ab9f-f8e1f7f27109","task_key":"default--validation--1591","task_revision_id":"1","upstream_id":"1591","short_description":"HellaSwag validation 1591","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to paint a door frame [title] Take the door off the hinges. [step] Take hold of the door from either side and lift up on it forcefully to slide it free from the attached wall hinges. Set the door aside where it won't be at risk of being damaged or getting paint on it.\",\"endings\":\"['Then, take your two inch wide flat-head screwdriver (a bathroom-size digital 12/17 \\\" pinhead screwdriver) and push it as far out from the frame as possible using proper technique. Allow the door to slowly loosen, getting as much of the paint off as possible.', 'Once the door is free from the wall hinges, you can start painting. [title] Paint the door as flat as possible with the paint gun.', '[title] Collect all the components and materials needed to paint the door frame. [step] A typical job to do before painting the door frame is framing, applying paint and putting on some faceplate coverings so paint has time to dry before you paint it.', \\\"[substeps] If you plan on painting the door the same color as the frame, simply leave it where it is. [title] Cover the door with a sheet of plastic if you can't remove it.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}