{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"560ef61f-97ac-59b0-b44d-1f65b08de52b","task_key":"default--validation--45153","task_revision_id":"1","upstream_id":"45153","short_description":"HellaSwag validation 45153","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to rust a nail [title] Sand off the rust proof coating. [step] Use a wire brush or sandpaper and rub over the surface of the nail. Continue to rub the sandpaper or brush over the nail until you scrape away the shiny finish.\",\"endings\":\"[\\\"[substeps] You can also use a metal or plastic scraper. [title] Choose a galvanized nail if you're doing a more extensive job.\\\", 'This will help keep the polish from chipping off. Sand to remove some of the rust and its protective coating.', \\\"Don't worry about getting scratches on the nail, since the rust will cover it anyway. [substeps] Use a 36-100 grit sandpaper.\\\", \\\"Let the sand dry completely before moving on. [substeps] If you don't have a wire brush, use a sponge instead.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}