{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"695c3790-7b36-5cab-a926-26c429b8abfa","task_key":"default--validation--9012","task_revision_id":"1","upstream_id":"9012","short_description":"HellaSwag validation 9012","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to keep a dog from chasing horses [title] Teach your dog to sit. [step] Before you introduce your dog and horse you need to be sure that you have taught your dog the basic commands, like sitting, that would allow you to keep them safely under control. Get on your puppy's level.\",\"endings\":\"['Then switch out puppy seats and allow them to sit with you. [substeps] Show your dog that you have taken your presence of affection into consideration.', 'Hold a treat close to its nose and move the treat up so that your dog begins to lower its butt. Once its butt hits the floor give it the treat.', 'Note how you might even need to have a friend teach your dog the commands. The best way to teach your puppy to sit is to use its training book or experiment with different commands.', \\\"Find its rear and hold it in your hand. Then, take the horse's rear and place it in your lap.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}