{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a0d6467-d441-5b30-abd7-fc5d5848a855","task_key":"default--validation--27521","task_revision_id":"1","upstream_id":"27521","short_description":"HellaSwag validation 27521","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"The little girl in the middle stirs the mixture in circular motion. Then she adds some melted butter to the mixture and stirs it again. then they\",\"endings\":\"['add some salt ans stir it again.', 'take their mugs and go outside and sit outdoors until the ice begins melting.', 'all try to taste the mixture but they have trouble and it tastes like liquid soap and then sour water.', 'put the mixture in the kettle and take the mugs down.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}