{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7375f63a-f8a6-585c-926d-7cec98259dbb","task_key":"default--validation--3289","task_revision_id":"1","upstream_id":"3289","short_description":"HellaSwag validation 3289","config":"default","split":"validation","body":"{\"activity_label\":\"Painting\",\"ctx\":\"An intro leads into a close up of a painting and a paintbrush swirling around and around. the hand holding the brush\",\"endings\":\"['dips it into paint and paints while the woman continues painting.', 'begins painting while standing next to a large piece of wood and pausing to wipe the brush on the canvas.', 'begins painting grass into the painting slowly down the picture.', 'is then seen using it to paint on the canvas causing it to move up and down and the logo continues flipping around but stops at the end.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}