{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"88a0891c-6407-5763-892c-ec458099e333","task_key":"default--validation--1103","task_revision_id":"1","upstream_id":"1103","short_description":"HellaSwag validation 1103","config":"default","split":"validation","body":"{\"activity_label\":\"Food and Entertaining\",\"ctx\":\"[header] How to do tuckpointing [title] Assess the situation. [step] The first step is to assess the area where damaged mortar will need to be replaced. [title] Set-up.\",\"endings\":\"['[step] Using appropriate equipment (ladders, scaffolding, aerial lifts etc. ), set up to reach the area safely.', '[step] Start the last step of the process by installing a basic mortar installation. If the damaged mortar is a good fit, then it will strengthen over time, allowing it to be placed.', '[step] Before you start, you need to decide whether you will approach the area or attempt to tackle it. Once you know where to determine the right path, you can go into the area to test it out, or simply just scout around the area.', '[step] If you are working in a apartment building with mortar breached, set the area up as soon as possible. Working at least 24 hours a day, according to your schedule and local laws.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}