{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bebd798d-59fd-5396-8f48-557b41e6e524","task_key":"default--validation--39495","task_revision_id":"1","upstream_id":"39495","short_description":"HellaSwag validation 39495","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to test your cat's intelligence [title] Hide your cat's favorite toy. [step] A cat's intelligence is similar to that of the average two-or three-year old child. One way to test your cat's intelligence is to hide one of her favorite toys behind a solid item such as a thick piece of paper or file folder.\",\"endings\":\"[\\\"The cat will find the toy and then go looking for it quickly so that it doesn't become a distraction. [substeps] Alternatively, you could hide a toy that is made of treated material rather than plastic or tin.\\\", 'Let her a get a good look at the toy first, then hide it behind the item. [substeps] It is best to do this activity in an uncluttered area where your cat can focus all of her attention on you.', \\\"[substeps] If your cat is familiar with solid objects, hide the objects in your cat's hiding spot. Your cat may instinctively pick them up and place them in her hiding location.\\\", 'Your cat will quickly search the paper to find the toy, and will display an immediate sense of joy and love for the toy. [substeps] If your cat is clumsy, she may consume objects that she finds and destroy them.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}