{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"799e1bbc-3a5f-526d-a04f-fa35b3cb345d","task_key":"default--validation--33855","task_revision_id":"1","upstream_id":"33855","short_description":"HellaSwag validation 33855","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to roast walnuts [title] Preheat the oven. [step] Turn on the oven and set it to 350ºf/180ºc. [title] Place walnuts on a cookie sheet or shallow baking pan. [step] Line the pan with parchment paper or kitchen foil.\",\"endings\":\"['Spray the pan with nonstick cooking spray. [title] Arrange the nuts on the parchment paper.', \\\"[substeps] If you choose to make a baking sheet, you'll need to line the bottom of the pan with tin foil before placing the nuts in the oven. You'll also need to cover the bottom of the pan with aluminum foil, since foil is not necessary when storing the nuts in the oven.\\\", \\\"Put the nuts on the pan in a single layer. [substeps] If you don't have parchment paper or foil you can just put the walnuts right on the pan.\\\", 'Preheat the oven to 475ºf. [title] Crush 2 large walnuts and crush them.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}