{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e9a0d9aa-d37b-5d89-b2a1-c2f88a664c9a","task_key":"default--validation--45099","task_revision_id":"1","upstream_id":"45099","short_description":"HellaSwag validation 45099","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make biscuit au chocolat [title] Crumble the biscuits. [step] In a large or medium bowl, crumble the 40 pieces of biscuit into small pieces. [substeps] You don't have to be exact about the number of biscuits you use, and you can use slightly more or less biscuits if you want, about five to eight more or less.\",\"endings\":\"['[title] Add the egg and breadcrumbs. [step] Beat the egg over low heat, stirring frequently to avoid over churning the egg.', \\\"It won't significantly change the results at the end. [title] Pour the can of sweetened condensed milk into the bowl.\\\", \\\"Use a fork to beat the biscuits into a smooth ball. Don't stir them in until it's totally smooth.\\\", '[title] Grate the butter into the bowl. [step] Place the butter in a bowl, then pour in some flour.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}