{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"630e1d69-96a3-558c-b31c-90534a0f6879","task_key":"default--validation--11783","task_revision_id":"1","upstream_id":"11783","short_description":"HellaSwag validation 11783","config":"default","split":"validation","body":"{\"activity_label\":\"Mixing drinks\",\"ctx\":\"This woman is showing her viewers how to make a chambord royale. first she\",\"endings\":\"['grinds it and adds salt and baking soda.', 'opens the bottle of beer that she had made and pours 1 perpatrick into the tiny glass with lime to continue drinking the beer.', 'fills her shaker up halfway with ice and then she adds vodka and chambord and she puts one ounce of both of them and then she adds a little bit of pineapple juice and cranberry juice.', 'demonstrates by mixing a mixture in a baking pan.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}