{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9096738c-a5ba-5ab1-8aae-0b204cc51e69","task_key":"default--validation--7650","task_revision_id":"1","upstream_id":"7650","short_description":"HellaSwag validation 7650","config":"default","split":"validation","body":"{\"activity_label\":\"Baking cookies\",\"ctx\":\"The two are shown mixing the bowl in turn. The two shapes the mix from the bowl into lumps and places them on a baking sheet. the two\",\"endings\":\"['mix the cookies over the oven with the arm of the baking sheet out and hand shake an the hands of the person holding the bowl.', 'are shown once again in several more sitka footage.', 'place the baking sheet in an oven.', 'are then seen blowing them onto a cake and on the edges on baking sheets.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}