{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bc7a5927-4f8b-534f-868b-ae2979ecb8dc","task_key":"default--validation--16735","task_revision_id":"1","upstream_id":"16735","short_description":"HellaSwag validation 16735","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to change a bathtub faucet [title] Turn off the water supply to the bathtub faucet. [step] This should always be the first step when taking on any plumbing project in your home. [substeps] Also have some rags and a bucket handy in case you need to clean up water quickly.\",\"endings\":\"['There is no need to wring out the toilet paper, since this will leave behind a film of residue. Towels can be used instead if you notice that liquids are pooling in the tub.', 'You might want to consider replacing the water supply with something new. It might be a good idea to take the drain line down when making small repairs.', '[title] Fill a bucket up with water and make sure you put a disposal container in the water. [step] Be sure to label the bucket with your name and address on the side, so you can easily know where to dispose of it afterwards.', '[title] Assess what type of faucet stems you have. [step] The faucet stems are the parts of your faucet that control the flow of water.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}