{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0342bdaf-12ba-5b70-8516-c444020b7d6f","task_key":"default--validation--16417","task_revision_id":"1","upstream_id":"16417","short_description":"HellaSwag validation 16417","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to replace hardwood floor [title] Put on gloves, knee pads, and protective eye glasses. [step] Make single cuts straight across the flooring that are about 1 foot (30 cm) to 2 feet (61 cm) apart. Be sure your cuts are perpendicular to the direction of the wood.\",\"endings\":\"['Make every cut across the entire floor in 4-5 even strokes. Repeat this process with all of the floors in your home.', '[title] Use a long piece of industrial door branch and exterior putty. [step] Take one end of the door branch and use the base strike direction to make chalk impressions on the floor.', \\\"It's okay if you cannot see the cuts clearly, as you cannot let the wood paint on the surface that you work in come into direct contact with your skin. [substeps] Before filling in the cuts, make sure you and your co workers are wearing the appropriate kinds of eye shields and protective eye glasses.\\\", 'Start on 1 side of the room and work your way systematically to the other side, spacing each cut about 1 foot (30 cm) to 2 feet (61 cm) apart. [substeps] Always keep your hands away from the blade of a circular saw.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}