The perils of Continuous Deployment

Continuous Deployment (CD), where changes are released several times a day, is popular among online game sites. IMVU for example, is a very strong champion of CD, and they use it to 50 times a day.

I used to use IMVU, and as far as I could tell, the releasing of changes was completely invisible to the user. But I have tried some other online games the use CD, and was not the case for them. What frequently happens is that you are doing the game thing, and a window pops up that says “The game has been enhanced. Please refresh your browser.” And when you refresh, the game takes two minutes to load in all of its assets, which really disrupts the game play.

An even worse situation is a Real-Time Strategy wargame that a friend was playing. The game deployed a change in the battle rules, regarding how much food had to be sent along with troops in battle. The change took effect immediately, right in the middle of a battle, and all her troops starved to death because the amount of food she had sent was no longer sufficient.

In case some of you are thinking “These are just games. What’s the big deal if your imaginary troops starve to death?” there are a couple of things to keep in mind. One is that there is a lot of money in online games, and things like this make your users get angry and go play somebody else’s game. Another is that the same thing could apply to any online system. Suppose someone is half way through booking a hotel room, and you deploy a change that alters the quoted room rate. They will not be happy when they get their bill, and it does not match what you quoted them.

It’s probably not possible to come up with a universal design pattern to make live deployments transparent to users, but as a minimum, the change should not completely lose the user’s state, should not require a bunch of time to reload assets that were already loaded, and should not change the outcome of transactions that are in process (like battles in a game, or hotel rooms that are being booked).


Leave a Reply

Your email address will not be published. Required fields are marked *