{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"hellaswag","formal_name":"HellaSwag","introduction":"HellaSwag asks which of four continuations naturally follows a sentence or a video caption. The wrong options are produced by adversarial filtering, so items that are nearly trivial for people remain hard for models.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://rowanzellers.com/hellaswag/","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"23dff695-4146-5075-862b-4cc05b2f3725","task_key":"default--validation--46887--variant-8be4352cf7e6","task_revision_id":"1","upstream_id":"46887--variant-8be4352cf7e6","short_description":"HellaSwag validation 46887","config":"default","split":"validation","body":"{\"activity_label\":\"Family Life\",\"ctx\":\"[header] How to paint brass [title] Remove the item if necessary. [step] Some brass objects, such as door handles, faucets, and fixtures will be easier to paint if you remove them from where they are. Other objects, like furniture, utensils, and lamps, are already portable.\",\"endings\":\"['Ensure that you clean the object thoroughly before you attempt to paint it, and if you make a mistake, contact a professional to help you do so. [substeps] Do not use anything hot or flammable to paint brass objects.', \\\"[substeps] If you removed any screws, nails, or other hardware, store them in a safe place so you can return the item after it's painted. It's also a good idea to test that your item is in fact brass.\\\", 'If you need to paint pots and pans, you may want to take apart the hardware and recycle the paint. [substeps] If your brass item is a jar or a bottle, take apart both before painting them.', \\\"[substeps] If you have a piece of brass that you think will be easy to paint, such as the old lamp shade, you should always let it sit in a bucket of water. [title] Cut a separate piece of wood as long as you don't cut yourself.\\\"]\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://rowanzellers.com/hellaswag/","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}