{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d6a10c7a-1bc6-51c1-9889-d95b5dc79373","task_key":"default--validation--43278","task_revision_id":"1","upstream_id":"43278","short_description":"HellaSwag validation 43278","config":"default","split":"validation","body":"{\"activity_label\":\"Computers and Electronics\",\"ctx\":\"[header] How to make oreo cookie balls [title] Place the cookies into a blender or food processor. [step] Don't take the cookies apart. Just place the entire cookie inside the blender/food processor.\",\"endings\":\"['[substeps] Make sure that you have coffee grounds or sugar on the side. For maximum, make sure that everything is touching the grinder.', \\\"Add the milk and water if you don't have a food processor or start the blender on low. [title] Blend for 1 minute, or until the mixture is stiff.\\\", \\\"If the cookies don't fit in your blender/food processor, then you will need to blend them in batches. [substeps] If you don't have either of those, place the cookies into a plastic, resealable bag.\\\", \\\"[title] Blend the ingredients on high until well blended. [step] If you don't have a food processor, go to a food processor and blend everything well.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}