{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a7f84bc3-f805-5477-8253-b8a873020021","task_key":"default--validation--7108","task_revision_id":"1","upstream_id":"7108","short_description":"HellaSwag validation 7108","config":"default","split":"validation","body":"{\"activity_label\":\"Elliptical trainer\",\"ctx\":\"An african american male wearing a blue muscle shirt and under armour sweat shorts is standing on an elliptical talking. although he\",\"endings\":\"['is talking he is also being held by being shirtless, and is showing off the number 678 alongside the elliptical opposite him that continues to display the number 779.', \\\"is talking to the camera and showing the body specifications of the machine it's powered down and you can see how it is slower.\\\", 'is the only one in the gym, there are two people standing behind him at the reception desk.', 'is wearing a tank top underneath he has a racquetball racketball racket and a black background.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}