{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7f7c4f0a-9cf2-59dd-bab8-5f3aac3a8b55","task_key":"default--validation--15117--variant-2653f400dd55","task_revision_id":"1","upstream_id":"15117--variant-2653f400dd55","short_description":"HellaSwag validation 15117","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to assess the safe firmness of an infant mattress to prevent asphyxiation [title] Gather your supplies. [step] Take your pen or marker, and your ruler. Place your ruler at the bottom edge of the carton.\",\"endings\":\"[\\\"Trace a line across the bottom of the carton starting from the bottom edge. Once you've got the line all the way across, take your paper or pencil.\\\", \\\"You'll also want your tape measure to be approximately 2 inches (5.08 centimeters) longer than the mattress. Write down the order in which you measured your mattress.\\\", 'Measure up an inch and a half (40 mm). Make a mark here and then draw a line running parallel to the bottom edge.', 'A ruler is a long, skinny, flat measuring tape measuring 24 inches (55 cm) tall. [substeps] Note that this rule applies to mattresses since the mattress is so large that it requires you to write your measurements in centimeters.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}