{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b140b6f3-ec18-5727-a2be-a48c8e40a77e","task_key":"default--validation--48063","task_revision_id":"1","upstream_id":"48063","short_description":"HellaSwag validation 48063","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to catch mesprit in pokémon diamond and pearl [title] Defeat team galactic. [step] Mesprit is not available to catch until you have proceeded through the story far enough to defeat team galactic by defeating cyrus and catching dialga/palkia. This will happen automatically in the story after you complete the seventh gym and raid the galactic headquarters.\",\"endings\":\"['[substeps] It takes several attempts before mesprit, or hamcatalysis, is available once you have successfully beaten all the characters and head to the final gym through the story theater station. Check to make sure you have chosen the correct game.', \\\"[title] Put a pokémon that knows surf in your party. [step] You'll need the surf ability to reach the verity cavern to trigger mesprit's appearance.\\\", 'By completing the next gym in the game, you can defeat mew and dialga in the second level. [title] Defeat the boss golem in the third gym.', 'Remember to stay on the top map, you will note the city of omega ruby, scour the desert and kill the leaders/officers in the sapphire gym. Now go back to miner village and grab a caffeinated formula.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}