{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"91da0c63-6cf6-51ac-98dc-ee1cae20d2e7","task_key":"default--validation--37283","task_revision_id":"1","upstream_id":"37283","short_description":"HellaSwag validation 37283","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to butter a brick [title] Grab the trowel. [step] Grab the trowel handle with your dominant hand. Your fingers should curl around the width of the handle, but your thumb should be extended along the length.\",\"endings\":\"['[substeps] Holding the trowel in this manner will give you greater control over it. Your wrapped fingers should keep the trowel sturdy and secure in your hands, and your extended thumb should make it easier to control the direction you move the trowel.', \\\"As your non-dominant hand covers the square and the two edges of the handle, use your other fingers to work the brick towards the center of the oven. [substeps] You'll see that the bottom section of the brick bends upward so that it's about halfway done.\\\", '[title] Use blunt or stiff strokes to work the butter out of the brick. [step] Keep applying slow increments with the tool until the butter has completely been removed from the granite.', \\\"How many inches is the handle wider than the tip of your palm? [substeps] The shank of your trowel is between the two front edges of the brick. Measure how thick you'd like your finish to be, then choose a medium length.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}