{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0197ce2c-059a-527d-af5b-f95d13d2dce4","task_key":"default--validation--7430","task_revision_id":"1","upstream_id":"7430","short_description":"HellaSwag validation 7430","config":"default","split":"validation","body":"{\"activity_label\":\"Making a cake\",\"ctx\":\"A large truck cake is shown, a smaller one next to it, and the toy truck is also there. the man\",\"endings\":\"['uses a hand squeegee to train the toy truck, and holds it extended for inspection until the toys are empty.', 'cleans the christmas tree, he picks up supplies, and makes sure the machine is running.', 'is now pouring more lotion onto his front in piles.', 'is back at the large light gray counter and in a sped up motion the man is shown with various different supplies as he assembles a very large truck cake that was previously shown.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}