{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3ae6c7e0-e07b-56d4-94ae-a0ed7f9a5f49","task_key":"default--validation--23822","task_revision_id":"1","upstream_id":"23822","short_description":"HellaSwag validation 23822","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to chop chocolate [title] Obtain your chocolate. [step] The type of chocolate depends on what you're making, and what type of chocolate you like. You can choose from dark, milk, or white chocolate.\",\"endings\":\"['Chocolate can be bought at almost any grocery store. Keep in mind that some types of chocolate are easier to chop than others.', '[substeps] Dark chocolates tend to be the easiest to combine, though milk and white chocolate are considered the rarest. Purchase chocolate from the dairy store in bulk if you need to store it.', 'For a very chocolaty flavor and creaminess, choose either cream, dark chocolate, milk or white chocolate. [substeps] When buying chocolate, visit a food store, a specialty supermarket, or online for the recipe.', \\\"[title] Spread chocolate across a towel, pot, or fire pit. [step] Put in the number of drops that i did exactly and don't over-estimate, or you may run out the door.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}