{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6ab55526-60af-5d24-9d6e-03b9ea981d3c","task_key":"default--validation--19386","task_revision_id":"1","upstream_id":"19386","short_description":"HellaSwag validation 19386","config":"default","split":"validation","body":"{\"activity_label\":\"Youth\",\"ctx\":\"[header] How to smoke in your house without your parents finding out [title] Find a private place to smoke it. [step] If at all possible, make it an outdoor spot. Part of the garage, or an area under your deck are both good choices.\",\"endings\":\"['Be sure to wear something you can walk on while you smoke. [title] Once you are outside, close the blinds and make sure there is adequate shade.', \\\"If an outdoor spot isn't an option, you can use the bathroom. [substeps] A bedroom isn't a great spot to smoke if you're trying to hide it.\\\", 'Run a quick internet search and see what you can find. Common places include your garage, back yard, backyard, and some house sheds.', \\\"That way, the smoke won't creep out after hours of sleeping. [title] If you're on a budget, don't borrow the money from someone other than yourself.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}