{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8f07c5bd-799a-5351-8fed-2a9109cca623","task_key":"default--validation--1736","task_revision_id":"1","upstream_id":"1736","short_description":"HellaSwag validation 1736","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make habanero hot sauce [title] Chop and sauté the carrots for five minutes. [step] Pour 1 tablespoon (15 ml) of olive oil into a large skillet and turn the heat to medium. Peel two large carrots and roughly chop them.\",\"endings\":\"['Set them aside and sauté the other carrots for five minutes. [substeps] You can use any kind of boiled water you like, but bottled or plain tap water is more likely to work.', \\\"[substeps] You don't need to mince the carrots since they're already finely chopped. You'll also need to add them to the pan.\\\", 'Add the carrots and sauté the carrots for five minutes. Stir them occasionally to prevent them from sticking.', 'Sauté them in the oil for five minutes over medium-high heat. Next, cook the carrots for another three minutes to prevent them from burning.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}