{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"84d14d59-4d32-5572-9830-30ccdab4dcbe","task_key":"default--validation--39497","task_revision_id":"1","upstream_id":"39497","short_description":"HellaSwag validation 39497","config":"default","split":"validation","body":"{\"activity_label\":\"Home and Garden\",\"ctx\":\"[header] How to fix a loose toilet paper holder [title] Examine the underside of the loose arm or escutcheon. [step] There should be a small hole with a screw inside it. This screw may or may not have a traditional \\\" head \\\" to it.\",\"endings\":\"['[title] Remove the screw if possible by grasping its edges between your index finger and thumb and gently twisting until it comes loose. [title] Take the screw down to the hardware store to get the correctly sized screwdriver to reinstall it tightly.', 'Pay attention to this hole so you can replace it. The screw may have had decals on it.', '[substeps] Before fastening the loose arm or escutcheon on top of the plastic cover, the gasket needs to be properly tightened. Check the screw for it\\\\'s \\\" head \\\" or a small screw underneath it that holds the key vent seam in place.', 'It usually travels down into the toilet paper holder but many of these holes are small enough to have a screw protruding out of them. [title] Shut off the flush valve to the toilet.']\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}