{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3cd97047-db32-59fd-b47f-6ebe0481a2c9","task_key":"default--validation--4936","task_revision_id":"1","upstream_id":"4936","short_description":"HellaSwag validation 4936","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to choose a mattress when you have back problems [title] Learn about mattress firmness ratings. [step] There's no definitive, industry-wide scale for mattress firmness. However, you'll find most mattresses labeled on a scale from extra-firm to extra-plush.\",\"endings\":\"[\\\"Mattresses rated for firmness are usually made of foam, cotton, and silk. [substeps] Bump mattresses feel more firm when they aren't moving.\\\", 'Each mattress has unique mattress recommendations, as do larger, heavier mattresses. Determine how often you will use a mattress and if you use it as soon as possible.', \\\"Your ideal mattress firmness should be somewhere in the middle of this scale. [substeps] Mattress firmness scales aren't uniform across all manufacturers, so ask a store salesperson if their mattresses typically run softer or firmer.\\\", '[substeps] Fitted mattresses typically have a rating ranging from these-0 to 255. Mattresses containing a higher rating will usually produce heavier mattresses, while mattresses with a lower rating will produce heavier ones.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}