{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3037e116-976c-546f-aa4b-a7905c0e04fa","task_key":"default--validation--7651","task_revision_id":"1","upstream_id":"7651","short_description":"HellaSwag validation 7651","config":"default","split":"validation","body":"{\"activity_label\":\"Baking cookies\",\"ctx\":\"The two shapes the mix from the bowl into lumps and places them on a baking sheet. The two place the baking sheet in an oven. The two girls hug each other. the two girls\",\"endings\":\"['place a brown paper bag over an oven.', 'flat ironed the baked sheets.', 'sit in front of the oven licking their mixing spoons.', 'grab plates and dish wash and put them in the dishwasher and return to flipping the cards.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}