{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"eae98784-4c67-5792-beab-44f642548a64","task_key":"default--validation--32625","task_revision_id":"1","upstream_id":"32625","short_description":"HellaSwag validation 32625","config":"default","split":"validation","body":"{\"activity_label\":\"Finance and Business\",\"ctx\":\"[header] How to make soup [title] Make a single vegetable soup. [step] Although other flavors are usually added to a soup focused on a single vegetable, such a soup is predominantly the flavor of the main vegetable used. Examples include : [substeps] Tomato soup or creamy tomato soup pumpkin soup cauliflower soup carrot soup cabbage soup sweet corn soup.\",\"endings\":\"['[title] Cool the soup in the stockpot. [step] Place the stockpot of soup back in the pot and turn it on its burner.', '[title] Make vegetable-based soups which contain a variety of vegetables and other ingredients. [step] Soups which mix veggies can come up with an enormous range of delicious flavors.', '[substeps] In other words, if you are planning on using a nonindian main vegetable, you should focus on making a single vegetable soup. [title] Check that your vegetables are ready for boiling.', 'Cauliflower soup is typically served warm or cold due to the sweetness of the vegetables. Chicken soup, on the other hand, is served cold due to the sweetness of the vegetables.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}