{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2a9ee037-5f98-5afe-b774-1b7f966172ce","task_key":"default--validation--39100","task_revision_id":"1","upstream_id":"39100","short_description":"HellaSwag validation 39100","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to annoy people in elevators [title] Push all the buttons. [step] When you get into the elevator, push the button for every floor. This makes everyone's ride on the elevator longer, if only for a few seconds.\",\"endings\":\"['If you want to take it a step further, look around questioningly at everyone in the elevator when no one gets off at the floors you pushed. [substeps] When someone comes in, say, \\\" i\\\\'ve got this, \\\" before you push all of the buttons.', 'This also makes the elevator go up faster. Be sure to be aware of the brakes that make cars skid as a result of getting into them.', \\\"[title] Talk to the person next to you. [step] While one person might be intentionally annoying another, nobody's in the elevator for long when they're already busy.\\\", '[substeps] When you climb up to the fourth floor, push the go button every 180 steps to exit the elevator. This will make it easier to move on from the elevator, especially if you got kicked out or if you got lost.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}