{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"39677a95-6ba5-5418-a672-594a54eacb17","task_key":"default--validation--30033","task_revision_id":"1","upstream_id":"30033","short_description":"HellaSwag validation 30033","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to dress like a hogwarts student [title] Select your house. [step] Choose one whose traits you admire, or take an online quiz. Every single student has a house that they belong to at hogwarts, so make sure you choose a house first thing.\",\"endings\":\"['At the same time, make sure your house is comfortable and doesn\\\\'t attract \\\" hogwarts \\\" with drugs, alcohol, or cigarettes, unless you are a real life person. [substeps] You can also draw on traditional family houses, such as the miniature beehive or baroque houses, to start.', \\\"You'll look like a crazy accused of murder in a haunted house, so it's better to be safe than sorry. [substeps] You can choose to live at a hogwarts castle if you'd like.\\\", \\\"Each house has its own house colors, which affect the colors of the uniforms. Gryffindor's colors are red and gold, slytherin's are green and silver, and hufflepuff's are black and yellow.\\\", '[title] Expect to pay more for your house, especially if you get into trouble in the process. [step] It may seem enormous, but it looks incredibly cool.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}