{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c23b63a5-cab6-51cb-8171-c86306d00b10","task_key":"default--validation--16498","task_revision_id":"1","upstream_id":"16498","short_description":"HellaSwag validation 16498","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to make your rabbit a leash [title] Use a harness, never a collar. [step] A collar risks strangling your rabbit or breaking its neck. Instead, use a figure-eight, figure-h, or vest harness.\",\"endings\":\"['This should be able to be made of multiple materials, however. The make the collar stick out about 1 inch (2.5 cm) at all times.', \\\"Remember, this is a safe, practice-free trick: this won't work with every rabbit. [title] Let your rabbit' run around' in a harness.\\\", 'If you do not wish to purchase a harness, you can make your own. [substeps] Figure-eight harness-this type of harness has two loops, one that fits behind the front legs and another that goes around the neck.', \\\"Don't tie up your rabbit while it walks, the collar will strike the rabbit in the leg. [title] Tie, in the same manner that you tie shoes.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}