{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c1c801e0-e776-5123-ba6e-80bffaba9476","task_key":"default--validation--37738","task_revision_id":"1","upstream_id":"37738","short_description":"HellaSwag validation 37738","config":"default","split":"validation","body":"{\"activity_label\":\"Personal Care and Style\",\"ctx\":\"[header] How to do hot stone massage [title] Find or purchase stones. [step] Stones used in this treatment are typically made of basalt, due to their ability to retain heat. The stones should also be very smooth, so they do not irritate the skin in any way.\",\"endings\":\"['[substeps] They tend to appear red or inflamed afterwards and can be painful and annoying. Gemstones ranging from yellow and greenish to orange are often used instead.', \\\"If you can't find basalt stones, however, smooth river rocks are fine. You can order a hot stone massage kit online from amazon or ebay.\\\", '[substeps] The stone hardness determines how many masseuses you should use. Larger stones with softer texture will require a larger masseuse.', 'You can purchase stones online or at big box stores. [title] Find a crystal or white coin with a pointed tip.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}