{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"01803ec4-6d2a-5de1-88f3-ff41d3fb5b58","task_key":"default--validation--12644","task_revision_id":"1","upstream_id":"12644","short_description":"HellaSwag validation 12644","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to cut a shirt [title] Get your shirt. [title] Cut a slit into the shirt on either side of the collar using some sharp fabric scissors. [step] Cut the slit along the collar seam to serve as a visual guide for the rest of your cutting project.\",\"endings\":\"['[title] Wash and lay the shirt (s) out. [step] Lay a sheet of newspaper in front of you.', '[substeps] You may notice the seam change while cutting, especially the collar. It will also be darker due to the stiffer, purer cut you will make.', 'However, do not cut at either the neckline or sleeve seam. [substeps] Use thin nails, making sure to cut all the way through the shirt, not just the seam.', '[title] Use fabric scissors to cut the collar off of the shirt. [title] Grab the fabric at the shoulder seam and pull it.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}