{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9ebacd4c-735e-5ccd-8829-0849f8c35181","task_key":"default--validation--27737","task_revision_id":"1","upstream_id":"27737","short_description":"HellaSwag validation 27737","config":"default","split":"validation","body":"{\"activity_label\":\"Holidays and Traditions\",\"ctx\":\"[header] How to create christmas games [title] Divide everyone into two (or more) groups. [title] Provide each team with a set of oversized mittens or gloves. [step] Have the members of each team stand in a single file line, with the first person in line wearing the gloves/mittens.\",\"endings\":\"[\\\"The other person then holds the hand of the first person and sets their mittens down into the snow. Each group should have 3 hands to hold the first person's mittens in place throughout the whole game.\\\", \\\"[title] Place a bowl of hershey's kisses on a table. [step] If you're the first person in line, run to the bowl.\\\", '[title] Each person should have sleeves to their elbow, typically full. [step] These sleeves must be 0.4 in (1.3 cm) in length; if it is too short, the team could die.', 'Have them stand in line in the front of the main lobby area (the character building area) until everyone reaches the end of their designated queue, then drop the first team member. Hold the second team member over a blanket or tablecloth until everyone has collected their scarf and gloves.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}