{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"39a9771b-5496-5eca-bef4-cdb2532b34c3","task_key":"default--validation--44568","task_revision_id":"1","upstream_id":"44568","short_description":"HellaSwag validation 44568","config":"default","split":"validation","body":"{\"activity_label\":\"Having an ice cream\",\"ctx\":\"The man boils milk and adds fresh vanilla and pours the solution into a bowl with other ingredients. the man\",\"endings\":\"['wellops up the finished skillet and puts it in the oven, cuts up the food along with the sliced fruit and uses a spoon to taste the cooked food.', 'joins other ingredients inside a bowl and pours it around before adding shots of rum and rum and mixing it together.', 'adds more water and squeezes it into the mug at the end.', 'cooks the custard and pours into a bowl and freezes to make ice cream.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}