{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3a67268a-7614-51a8-8ad7-11883eaf7c91","task_key":"default--validation--34420","task_revision_id":"1","upstream_id":"34420","short_description":"HellaSwag validation 34420","config":"default","split":"validation","body":"{\"activity_label\":\"Installing carpet\",\"ctx\":\"The group of men put the carpet on the floor, they measure it, cut it and taped it. when the new carpet\",\"endings\":\"['is on the floor, they start to scrub it to remove any residue.', 'is on the ground they put it in the laundry closet.', 'is properly installed, the men vacuum the carpet while the lady walked in the living room.', 'is on the floor they puts the corners together then started putting the tiles down all along the floor.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}