{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4fc564f7-0dea-5b0e-91f0-ca828df375a4","task_key":"default--validation--41746","task_revision_id":"1","upstream_id":"41746","short_description":"HellaSwag validation 41746","config":"default","split":"validation","body":"{\"activity_label\":\"Preparing salad\",\"ctx\":\"A man and a woman are standing in a kitchen behind an island with fruits on them. Once the male finishes talking, the woman begins grating a vegetable and cutting an orange in half. she\",\"endings\":\"['then mixed the three ingredients that she had created together in a cutting block and once it was complete, she proceeded to cut them up and mixed the mixture into it.', 'then speaks to the camera while measuring and sightening the vegetables, while still speaking to the man.', 'then squeezes the orange into a clear glass bowl and seasoning it and mixing it together with the previously cut up ingredients.', 'grabs a large knife and starts slicing the vegetables over and over again, when finished, she dumps old stainless steel in the garbage.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}