{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8df2fbb7-d0de-53d4-b096-80beebc01ff6","task_key":"default--validation--26898","task_revision_id":"1","upstream_id":"26898","short_description":"HellaSwag validation 26898","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to find someone's probation officer [title] Confirm the full name of the person on probation. [step] The guy you know as bubba jones may actually be named john smith iii. Many people use nicknames or aliases in daily life, sometimes even just their middle name instead of their given first name.\",\"endings\":\"[\\\"[substeps] Also keep in mind that if you didn't know mr. jones before on social media, or in person, you are not likely to know anyone else by his first name.\\\", \\\"Forcing a name at the very least might fool a person. [substeps] You also want to make sure the name doesn't look or sound too different from someone else's name.\\\", \\\"[substeps] If you know the name of that person, you know the full name. If you don't know the full real name of that person, write that name down.\\\", \\\"[title] Determine that the individual isn't in prison. [step] Sometimes people bounce back and forth between prison and probation.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}