{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ccf2f7d4-1a0e-5a6c-b5ba-7823f1db3338","task_key":"default--validation--9007","task_revision_id":"1","upstream_id":"9007","short_description":"HellaSwag validation 9007","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to become an fbi agent [title] Make sure you meet the minimum requirements for application. [substeps] You must be under 37 years of age when you're hired but at least 23 years of age. Have a driver's license in a us state.\",\"endings\":\"['Be a citizen of another country with no previous criminal history or any criminal convictions within the last three years. You must also have a social security number but no previous criminal convictions.', 'Be 21 or older when you apply for licensing. You must be a resident of a us city or county.', 'Have a valid valid federal or state-issued id (usps). Notify your employer and other consumers as soon as possible, which may affect their decision to hire you.', 'Have a minimum of a bachelors degree from an accredited four year university be physically able to complete the training and execute the duties. This includes the ability to fight, complete fire arms training, and other duties as required.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}