{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c5357e05-2920-5879-8974-cd4121e3b7a7","task_key":"default--validation--16014","task_revision_id":"1","upstream_id":"16014","short_description":"HellaSwag validation 16014","config":"default","split":"validation","body":"{\"activity_label\":\"Getting a piercing\",\"ctx\":\"An introduction comes onto the screen for a video about a piercing method. A woman sits and waits as she prepares to get a piercing in her cheek. she\",\"endings\":\"['will wrap her finger around the butt of the piercing to keep it out of sight and sits as she gets the piercing out of her nose.', 'lays down and receives the piercing in her cheek, then sits up to wipe her face.', 'does a small shiner on her cheek after the procedure makes it into place.', 'has a piercing cabinet, but it looks extremely ornate at the moment.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}