{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"03d6e2fe-5556-5004-86a5-07702701070d","task_key":"default--validation--38714","task_revision_id":"1","upstream_id":"38714","short_description":"HellaSwag validation 38714","config":"default","split":"validation","body":"{\"activity_label\":\"Sports and Fitness\",\"ctx\":\"[header] How to perform the ten basic okinawan karate moves [title] Find an open space to perform the following basic karate moves. [step] Make sure to also wear clothing that is comfortable and flexible. [title] Set up a basic stance.\",\"endings\":\"['[step] Stand with feet shoulder width apart and shoulders level. Point your toes towards the sky, or in another location, and place your left hand slightly in front of you, at the shoulder height you want.', '[step] Stand up in basic stance which involves having your feet spread out and your hands in fists next to each side of your waist. [substeps] It is important to make sure your thumb is on top of your fist at all times.', '[step] Find a space in which to stand and place your feet in a comfortable position. [title] Bend your knees and place your right foot in the middle of the stance.', '[step] Hold your arms diagonally with the bottom elbow pointing slightly forward (an open posture will make your stance harder to force). [title] Insert a knife at different points near the elbow.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}