{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a506b3fe-ffc7-58f5-bfab-5ef3ef57ed10","task_key":"default--validation--41385","task_revision_id":"1","upstream_id":"41385","short_description":"HellaSwag validation 41385","config":"default","split":"validation","body":"{\"activity_label\":\"Work World\",\"ctx\":\"[header] How to become a locksmith [title] Research the career. [step] As a locksmith, you may install and repair locks for both buildings, automobiles, and safes. You may also need to make duplicate keys and offer forms of related assistance.\",\"endings\":\"['[substeps] Many locksmiths work in a shop during normal business hours, but you should also expect to visit customers on-site when installing locks or assisting someone locked out of a vehicle or building. Trainees usually start at minimum wage, but your earnings will typically grow quickly as you gain experience and demonstrate productive skill.', 'Examine the exam schedule and topics specific to the particular job or project you are applying for. [substeps] Some colleges and universities offer practical mock-exams that cure exam symptoms, preventing you from perfecting your own craft.', '[substeps] You may want to go online and find keywords and other resources about what locksmiths in your area study. If any information is helpful, talk to your local police department or civil rights office.', '[substeps] Look for a locksmith that specializes in lockpicking. They will often have a certification program or accredited move or license, such as a united state locksmith registration agency.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}