{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8d86b431-5369-5c30-ae7a-90a13df3c4d2","task_key":"default--validation--43680","task_revision_id":"1","upstream_id":"43680","short_description":"HellaSwag validation 43680","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to get rid of wasps in your house [title] Gather your materials. [step] This is easy to make and can be put in the corner of the room that has a wasp problem. [substeps] Scissors empty plastic bottle (2l) tape bait (watered down dish soap ) [title] Remove the bottle cap and cut through the bottle a third of the way down.\",\"endings\":\"[\\\"[step] Be sure to do this carefully with your scissors. You can also use a razor blade if the scissors don't seem to be cutting through the bottle.\\\", \\\"[step] This lets the wasps have the chance to escape. Be very careful here--wasps like to drown in water for awhile, so don't have your black bottle either.\\\", '[title] Pour boiling water into the bottle through the piercing holes. [step] Leave just a little room, within there may be a nest.', '[step] Your box may have a cover for it; simply remove the lid and cut through that half. [substeps] It is important to go slowly as you are cutting.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}