{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"40953c02-8b61-52d4-8a9b-86e84312fb4c","task_key":"default--validation--31153","task_revision_id":"1","upstream_id":"31153","short_description":"HellaSwag validation 31153","config":"default","split":"validation","body":"{\"activity_label\":\"Baking cookies\",\"ctx\":\"The mixes again and finally puts another small amount of ingredients in the bowl. She mixes a third time and adds chocolate chips, finally mixing one more time. the oven peeps\",\"endings\":\"['and she prepares a pan with cookie dough.', 'down the hall to turn off its lights before they shut off, doors shut and windows open in the dark.', 'out to see what is left to bake before the woman jumps back into the hot iron.', 'out of the oven and lets out some white smoke to celebrate the recipe.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}