{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"df60ace7-f56e-5726-ab6b-f2b1f711ecf3","task_key":"default--validation--8565","task_revision_id":"1","upstream_id":"8565","short_description":"HellaSwag validation 8565","config":"default","split":"validation","body":"{\"activity_label\":\"Sharpening knives\",\"ctx\":\"Three people are standing behind a counter. A man puts pennies on a block before rubbing his knife on the block. he\",\"endings\":\"['cuts the block of blocks in half.', 'dips the knife through the block then dropping it on top of it.', 'cuts along the sides and places the mix in the mixer.', 'holds a knife sharpener and rubs the knife up and down the sharpener.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}