{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7f362338-44e3-5344-9737-dc580b67e06c","task_key":"default--validation--36587","task_revision_id":"1","upstream_id":"36587","short_description":"HellaSwag validation 36587","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to dunk an oreo cookie [title] Obtain a box of oreo cookies from the local grocery store or gas station. [step] Make sure they are the original cookies and not the new vanilla oreos. While at the store, pick up a gallon or half gallon of milk.\",\"endings\":\"[\\\"Use a pair of shallow feet to lift up the box to remove the cracked cookies. [substeps] If you can't get any oreo cookies, you can also buy regular oreo cookies instead.\\\", \\\"( 2 l-4 l) if you think you are going to spill much of the milk, go with the gallon (4 l), but if you are a neat and tidy eater, go with the half gallon (2 l). If you already have milk, make sure it's fresh.\\\", \\\"Do not remove the cookies when the milk is poured in, though; you'll be holding them for a long time and using up the milk in the process. [substeps] Follow the directions on the box of cookies for the correct amount of time.\\\", 'Place the oreo cookie in the milk container and then dunk your cookie, immersing it in the milk. Allow your cookie to soak in the milk overnight to allow the oreo to soften (waiting too long for it to absorb).']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}