{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"73533c74-e960-5e52-b85a-c36ac9b864b3","task_key":"default--validation--3924","task_revision_id":"1","upstream_id":"3924","short_description":"HellaSwag validation 3924","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make an easy bowl for smoking [title] Gather your materials. [step] For this project you'll need a 2-litre plastic bottle, aluminum foil, a pen, tape, a lighter, a small plastic pill bottle, and a cigarette. [title] Put a hole in your bottle for the stem.\",\"endings\":\"['[step] Burn a hole through the 2-litre bottle with your cigarette wherever you want your down stem to go in. This is usually closer to the bottom, anywhere from one third to one half of the way up the bottle.', '[step] Poke a hole in the bottle in the side where the stem and lid were, and put your cigarette in it. Poke through the wrapper or lid too.', '[step] Use water or oil to make a small hole, at least 4-5 centimeter (1.1-1.2 in) in this will be enough for your cigarette to be put through. [title] Cut a small hole into the bottom of the bottle to connect it with its stem.', '[step] Crack open a small hole in the side of your plastic bottle that the pen cuts into to allow for air to be released from. Tie a thin rubber band around the bottle and insert the pen through to find the stem.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}