{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"89f14705-25aa-5ca1-ad4c-f9a28e0d1da6","task_key":"default--validation--26369--variant-a9a563bedcdf","task_revision_id":"1","upstream_id":"26369--variant-a9a563bedcdf","short_description":"HellaSwag validation 26369","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to unlock a door [title] Use a credit card on a spring lock. [step] Though it won't work for deadbolts, it will work for spring locks (the kind that's slanted and a part of a handle). Grab one that's cheap and you don't care about--if it breaks, no harm, no foul.\",\"endings\":\"['A bendy, laminated card works best. [substeps] Force it between the lock and the frame, flush against the frame.', '[title] Push your finger down on the latch to open the door. [step] If you have a mini spring lock, push your finger down through the top of the lock to open it.', '[title] Jump into the lock and press the in button. [step] Do this for a quick, forceful push, which the lock will reach and swing open once unlocked.', 'Once you do that, you can rip the lock right out of the door. If you do not have a credit card, borrow from a friend, or rent one from a local hardware store.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}