{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"63d003ad-dae9-5f8c-a3cd-06f459466be4","task_key":"default--validation--20287","task_revision_id":"1","upstream_id":"20287","short_description":"HellaSwag validation 20287","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to act when you find gas leaking in the kitchen (usa ) [title] Determine the danger. [step] The safety of people is always the first thing to consider, so heed the following first of all : [substeps] If it is difficult to breathe or the smell is very strong or you can hear a loud sound of escaping gas leave the kitchen immediately and get any other people out of the building. Do not turn on light switches in or around the kitchen.\",\"endings\":\"['Put on protective clothing and a respirator if possible. Consider keeping a friend or family member home with you while you search for gas.', 'Do not eat or drink, and do not go in the kitchen if unsafe. [title] Separate all of the people in your home.', 'Do not turn any electrical switches on or off, including doorbells. Do not use matches or any other naked flames.', 'Stay near the side of the kitchen where gas leaks are more likely to happen. Check to make sure the door is closed and no one is around.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}