{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f6fce037-841b-5763-8520-91cc02f2a077","task_key":"default--validation--14377","task_revision_id":"1","upstream_id":"14377","short_description":"HellaSwag validation 14377","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to make a pancake mix container from a recycled ketchup bottle [title] Scrub the ketchup bottle thoroughly. [step] Make sure you are completely finished with the remaining ketchup and proceed to give the bottle a good scrub. You don't want any remaining ketchup hiding inside the bottle or else you will end up with ketchup pancakes! [substeps] Rinse the inside of the bottle with warm water and then add mild dish soap and scrub.\",\"endings\":\"['When scrubbing the inside, you want to be sure you rinse most of the residue away. [title] Replace the cap of the ketchup bottle.', 'Scrub the inside for at least 30 seconds if doing all the lather yourself. Rinse thoroughly and repeat if necessary until the counter top is clean.', 'Use a thin, long scrubbing brush to get the inside of the bottle. Rinse soap thoroughly and allow the bottle to dry.', 'This helps even the application of the ketchup in the wash cycle. [title] Shake it vigorously and assess the consistency.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}