{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4fe2cbc6-df9c-5dfa-b68b-6b8edc51e5e9","task_key":"default--validation--42873","task_revision_id":"1","upstream_id":"42873","short_description":"HellaSwag validation 42873","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to handpaint knobs for a room [title] Prepare the knobs. [step] Unfinished knobs are available in a variety of materials, each requiring a different preparation. Wooden knobs will need to be sanded lightly with 220 grit sandpaper to remove any surface irregularities.\",\"endings\":\"['You may prefer not to use primer if you like the grain of the wood to show through a bit. Metal knobs will need to be primed with a spray metal primer (i.e.', \\\"[title] Select a stain-resistant varnish. [step] For steel knobs, you'll need to use caulk or oil for the stain.\\\", 'They should be sanded with 120 grit sandpaper in order to prepare them for painting. [substeps] Look for knobs made from stainless steel or sterling silver.', 'Single edges and edges should be re-sanded lightly to keep the wood from splintering and leaving behind scratches. Primer primer dries when primed, but you can skip this step if you want the knobs to last longer.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}