{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2c052c2d-c116-50c4-bc4f-b07b87490fe7","task_key":"default--validation--35218","task_revision_id":"1","upstream_id":"35218","short_description":"HellaSwag validation 35218","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to keep a cat from running out the door [title] Use an alternate entrance and exit. [step] If your cat is constantly hanging out by the front door, waiting to make its escape when some unwitting person opens it, try exiting out a different door. For instance, instead of going in and out through the front door, try using the back or garage door.\",\"endings\":\"[\\\"Another good option is to exit and enter via a door with an antechamber. After passing through the first door, close it firmly behind you and look about to ensure your feline friend hasn't followed you.\\\", \\\"Another option is to bypass your house's rear entrance and enter the garage door, opening it by turning the opposite way. This will block the exit, while also allowing the cat to do the same.\\\", 'Be aware of this rule before making a move toward your other door; your cat will pounce on you and use this as an opening to escape. [substeps] Avoid allowing your cat to enter one door first.', 'Of course, this will only take up half the room in which your cat can go and should be considered a safe option. However, remember that your cat has a natural instinct to leave that door open if something happens.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}