We're currently in process on a monthly update for our flagship product and a Beta release for our new Electronic Medical Records product(see the 6N Systems site if you're interested in learning more). As an ASP, we make data updates to centralised (yet separate) db servers, then push webservice changes, then finally update a file patching application that will load the any changed client functionality on the next user login.

This model has been in place for quite a while, and it's been somewhat successful over the years. I say "somewhat" because as we've grown, we have more clients, more patch requests, and at least one problem to be solved each time we do a push.

As I see it, the problems that we have can include:

• Odd permissions issues where files pushed for client delivery are marked as ReadOnly. Our automation software doesn't attempt any file wipes in this instance, as it could potentially backfire if not written properly (think of accidentally deleting reports saved by your customers).
• Hot Fixes accidentally including full update scripts, not specific patch scripts.
• Customers changing our access permissions so that we can't do the updates that they expect us to do.
• Customers blindly changing permissions of components and functional elements of the system despite regular reminders not to change the policies.

It's an interesting balance when dealing with the security side of things: we're striving for usability and efficiency, while security admins are protecting the fort. When customers affect a login for us, it's bad, but not critical: an email usually fixes the problem, or the update is held until we can get access to all systems. Locking out programming functionality, however, can cause untold issues for customers, including lost revenue or fines because of incorrect billing. There's lost time on our side as well, as we usually are involved in troubleshooting the issue, and having to face the "we didn't touch the system" time and again. For better or worse, there is no completely automated cure for this level of the human element.

However, we can help our own processes through automation. Our development plans include moving to Team System, which will hopefully help in the deployment process (I'm already sure it's better than Source Safe). Of course, it will also improve quality control with easy management of functionality unit tests, which will hopefully reduce some of the hotfixes. In my experience, hotfixes usually patching something where the most recent version wasn't correctly grabbed, or new functionality breaks something apparently unrelated.

I'll keep you posted when we make the move. Being in the middle of a Beta push doesn't help the move to Team System. Until then, I'm going to anticipate a long night a month for each release, and the loud rings of the customer hotline for the days that follow.