{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31e24c59-2eff-5685-b15f-964378f3900a","task_key":"default--validation--47574","task_revision_id":"1","upstream_id":"47574","short_description":"HellaSwag validation 47574","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to make a severed hand snack tray [title] Decide whose hands you want to use for this project. [step] Invite some friends over and make a party out of it. Just remind them to wear old clothing that they don't mind damaging.\",\"endings\":\"['[title] Cover your floor or work surface with plastic sheeting or a canvas dropcloth. [title] Grab a bucket that\\\\'s large enough to completely submerge your model\\\\'s hand along with a 2 \\\" (5 cm) section of the wrist.', '[title] Cut out a square piece of white foam to make the base of the tray. [step] Use a butter knife to cut it out on a sheet of plain paper.', \\\"Maybe party hats, old gloves, or old diapers. If you plan to serve food for the party, remind the friends that it would be in their best interest to wear old clothing that they don't mind damaging.\\\", 'Say: \\\" you guys are last to wear your gloves. This is so your hands will stick out and open.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}