{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11e26546-2c46-5637-9137-2cfacc06cf9f","task_key":"default--validation--35416","task_revision_id":"1","upstream_id":"35416","short_description":"HellaSwag validation 35416","config":"default","split":"validation","body":"{\"activity_label\":\"Cleaning sink\",\"ctx\":\"A woman is shown walking around a kitchen turning on a faucet and l letting the sink fill up with water. She is shown wearing gloves, spraying cleaner around the sink and scrubbing with a brush. she\",\"endings\":\"['finally uses a sponge to thoroughly clean a sink using a squeegee to clean the water and sink.', 'smiles at the cleaning product and pans out to the label being shown.', 'continues talking to the camera and begins turning off the water and returning the sink into the same position.', 'sprays more product around the sink and then proceeds to clear it with a sponge.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}