{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"54b75dec-83e1-5390-b663-254c61a9e3fa","task_key":"default--validation--30024","task_revision_id":"1","upstream_id":"30024","short_description":"HellaSwag validation 30024","config":"default","split":"validation","body":"{\"activity_label\":\"Pets and Animals\",\"ctx\":\"[header] How to sex tortoises [title] Pick up the tortoise. [step] Hold the tortoise in both hands, being careful not to drop it. Hold the tortoise up high enough to be able to see underneath it.\",\"endings\":\"['Hold the tortoise up close to your body so that it is not forced against you. Close your arms around its neck and press it into your chest.', 'Lower yourself until its head is touch your lower ribs. [substeps] Be careful! Tortoises hide their genitals.', 'Do not flip the tortoise on its back. This is very stressful for the animal.', \\\"Explain to the tortoise what you are doing, and gently start stroking the tortoise's belly. [substeps] Speak softly in such a low voice that the tortoise won't notice you are petting it.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}