{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b6a14af5-303b-5062-850d-dceeda7b276d","task_key":"default--validation--39441","task_revision_id":"1","upstream_id":"39441","short_description":"HellaSwag validation 39441","config":"default","split":"validation","body":"{\"activity_label\":\"Installing carpet\",\"ctx\":\"A person on hands and knees hammers nails into a long wooden strip on the floor next to floorboards. A person pours glue on the floor underneath the carpet. the person\",\"endings\":\"['holds large metal ball in front of the nail and holds the pile up with the ball sticking out.', 'begins to use adhesive and carpet laying tools to steam and adhere the carpet to the floor.', 'installs shingles to a floor crack surrounding the shingles.', 'opts on installs custom welds on the surfaces of the floorboards.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}