{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"eada6997-9b48-5daf-bb22-f8da24fb4538","task_key":"default--validation--3207","task_revision_id":"1","upstream_id":"3207","short_description":"HellaSwag validation 3207","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to make a smoker grill [title] Fill a large bucket with water, and pour in approximately 5 lbs. [step] ( 2.27 kg) of hard wood smoking chips. Ensure that the chips are completely submerged in the water, and allow them to soak thoroughly for at least 1 hour before placing them in the smoking grill.\",\"endings\":\"['If you add more water, the water will reflect off of the wood chips rather than soak them. Stir vigorously to ensure that the wood chips do not spill over the edge of the bucket into which you are placing them.', '[title] Remove the grill grate and grill lid from your kettle-style charcoal grill. [title] Open the grill vents, and place 4 to 5 large handfuls of natural lump charcoal on 1 side of the grill grate.', '[substeps] In addition to actually encouraging the smoke to escape more easily, you will also be increasing the minute smoking time. This ratio will encourage smoke to escape more freely once the meat is seared.', '[substeps] Also, scrape any ash from the wood so that it is not standing on top of the wood chips. [title] Light the grill at 180ºc/375ºf.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}