{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ee301227-ddd8-507c-890e-f4d3367af8b8","task_key":"default--validation--24767","task_revision_id":"1","upstream_id":"24767","short_description":"HellaSwag validation 24767","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to separate sand and salt [title] Gather your supplies. [step] Because this is such a straightforward experiment, you won't need any lab gear or equipment. This is a cheap experiment.\",\"endings\":\"['Dig a hole about 2 by 3 feet (1.2 by 2.4 meters) deep and a few inches higher than the aquatic floor of your container. Shovel into the hole and the sand and salt will stand on their own.', 'There are a good number of items you will need for this experiment. These items can be either sanitation/extraction supplies or : [substeps] A kind of clean container to collect dirt, sand, or salt.', '[substeps] Make sure you have all the materials needed in order to separate your sand from the gravel, but be sure not to misuse any parts. Consider purchasing chemical salt or peroxide hybrid (formerly called \\\" hydroliner \\\") for your saltwater mix.', \\\"Here's a few things you'll need : [substeps] Salt. Most households have table salt in the kitchen.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}