{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a545eecc-7cfc-5477-b19d-ce6255bf567b","task_key":"default--validation--8005","task_revision_id":"1","upstream_id":"8005","short_description":"HellaSwag validation 8005","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to mingle with strangers at parties [title] Introduce yourself. [step] Select a party-goer and approach them. Tell them who you are.\",\"endings\":\"['You might, for instance, say, \\\" hi, i\\\\'m mike, \\\" then shake hands with them. Ensure your handshake is neither too limp nor too hard.', \\\"Even if they don't understand you yet, they're going to remember your name. Roll a paper wrapper one or two times and drop one on their shoulder.\\\", 'Say something like, \\\" hi, i\\\\'m jessica, but you can call me meg. \\\" if a stranger asks you for the name or has any questions, just say \\\" hey, i\\\\'m jessica, and i\\\\'m new here.', '[substeps] Say \\\" hi \\\" to the person. You can say something like \\\" hi i\\\\'m jane smith and i\\\\'m interested in going to a party.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}