{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1d0b27c6-b7ec-56df-b55d-619e9db477cd","task_key":"default--validation--21854","task_revision_id":"1","upstream_id":"21854","short_description":"HellaSwag validation 21854","config":"default","split":"validation","body":"{\"activity_label\":\"Making a lemonade\",\"ctx\":\"At an oven, she prepares simple sugar with water and sugar. She mixes all of her ingredients in a pitcher to make lemonade. she\",\"endings\":\"['shows other lemonade recipes.', 'walks outside to fetch the lemonade.', 'spreads the butter with milk, pouring it into jugs and serving it.', 'adds two bottles of apple juice and some eggs to the pitcher.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}