{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7e98492a-b41a-5c05-b3d4-facfed8c81f7","task_key":"default--validation--46788","task_revision_id":"1","upstream_id":"46788","short_description":"HellaSwag validation 46788","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to kill woodrats [title] Take the trap out of the packaging and unhook the arm bar. [step] Use a set of pliers to remove the staple that holds the arm bar in place on the trap. Move the arm bar so that it's hanging off of the trap for the time being.\",\"endings\":\"[\\\"Gently lower the arm bar out of the trap. If it isn't hanging off easily, twist and lift the arm bar.\\\", 'If a tiny black bubble bubble is forming on the bait, the lure should fly off. Instead, it will land on the bait.', '[substeps] Snap traps are the most effective form of trap when used correctly. If your trap comes with directions, make sure you take the time to read them.', '[title] Use an ice pick to remove the arm bar. [step] Hold the ice pick between your fingers.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}