Software quality is the great white whale of the software industry. We often seek it, but are totally unprepared to do anything about it. As in the tale by Melville, we can point out our Captain Ahab’s who are driven towards the extremes in ensuring quality. We also have the rest of the world that gets by not even thinking about it, except to say we know someone who knows someone that heard someone else talk about it in the dark corners of tavern once. It is one of the most critical facets of a successful software project, yet most teams are willing to cut corners in quality to meet other goals. In my experience, quality is often the second area where team resources are cut, right after documentation.
I bring this up now, because I have recently installed Windows 7 on a new machine, and with it, all of the software packages. As I installed the applications, I would launch them to test that it installed correctly. Making things more interesting was the release of Hearts of Iron III on August 7th, 2009, and the every Tuesday patch to World of Warcraft. The combination of all these events game me a unique opportunity to relive the first impressions of installing a new application. The result of which was quite surprising.
I have been covering my installation experience of Windows 7 in posts here, here, and here. The short answer is that Windows 7 has been a rock solid release. It reminds me of the Windows XP release – it just works. Once Windows is installed, comes the Office installs, in this case it was Office 2007, One Note 2007, Visio 2007, and Project 2007. As with Windows, it just worked. No issues at all. The came Visual Studio 2008, and again, a smooth install process and everything just worked. Of course, just working after an initial install is the easy test case. Once I re-configured Outlook, and my larger than 100gb PST file was my new default, then Outlook reverted to its true nature – crashing every so often. Other than Outlook, the only other issue I ran into was some minor Visual Studio 2008 conflicts between some plug-ins. All in all, a very smooth, albeit long, experience.
Then came Hearts of Iron III (HOI3) and World of Warcraft (WoW). WoW installs fine, but since they have not yet adapted to the system folder protections in Vista and Windows 7, so the solution seems to be installing into the public folders where the WoW directory will not be treated as a system folder. HOI3 runs into a similar issue, but only if you try to modify the data files – as you would to modify the game play. HOI3 is solved more easily by just changing the permissions on the folder to FULL for the logged in user. WoW might actually be solved the same way, but I have not bothered to try. Maybe I am old fashioned, but I do expect things to install and uninstall cleanly. I know. I am delusional, but I still hold out hope for true XCopy deployment.
Why all this talk about installs and these specific products? The answer is simple. I think it reflects on the quality of the software, and the testing done. However, I don’t want to use just the installation experience as the benchmark. That wouldn’t be truly fair as it is a cooperative effort of an ever changing relationship between operating system and application program. No, I will speak to the application itself.
When I start Word, it works. When I create a new document in Word, it works. No matter how complex a document I have ever produced, it has always worked. The same is true for Excel, Power Point, and Project. They just work. Outlook usually works, and the same is true for One Note. Visual Studio, yep, it just works. However, the same cannot be said for HOI3 or WoW.
Take WoW as an example. WoW has to be one of the most complex applications ever written. It supports hundreds of thousands simultaneous users, with clients on multiple platforms, and from everywhere in the world. It has to be one of the best scaled applications ever. Blizzard has a long history of producing very popular games, and these games leverage much of what a computer has to offer. To constantly address the WoW world, Blizzard created a system in which they update every Tuesday – so called Patch Day. Sometimes these patches are small, nearly trivial issues. Other times, they are truly epic in scale. However, there is one common thread to all patch days, they always go wrong. Hardly a Tuesday comes where one or more of the game servers is up on time, and functioning properly. It might be increased lag, or new treasure that is worth way too much or too little, or suddenly one group of player characters is significantly more powerful or weaker than the others. It doesn’t quite matter what the issue is, just that there is always an issue. The more complex the patch, the more significant the follow-on issues. While it may be acceptable for some for accept these issues, I don’t. I believe it is a sign of flawed software quality process. With years of experience, they should have a very good automated test system, combined with a broad range of human testers. I do not expect Blizzard to catch every issue, but I do expect that most Tuesday’s would patch without any follow-on issue. The post-patch problems should be the exception rather than the norm. We are talking about 8-10% of monthly play time impacted by these issues, and that is time paid for by customers that are involuntarily converted into beta testers. While I have no insight into Blizzard’s management process, and I actually have great admiration for their development teams, I do believe that they choose to sacrifice quality in order to achieve a release date, and that is a choice that I believe is bad for business in the long term. Follow this up by the perception (it has not been reality in my experience) that Blizzard is unresponsive to the players, and you have the customer base waiting for something worth playing other than WoW. That is not the position I would want my products to be in. This is similar to the position Microsoft has been in concerning Vista. Despite the improvements of Vista Service Pack 2, many users would consider something else if it could do what they wanted.
Paradox is in a slightly different position. They are very responsive to their user base. In fact, they are one of the most responsive I have ever seen. They are quick to patch issues, and will frequently address those issues within days or weeks. However, they suffer from another classic Microsoft perception problem – The Three Version Rule. The Three Version Rule states that you do not use the product until the third version (patch, service pack, revision, etc…) – period. Nearly every company I worked at until just after Microsoft XP came out, followed The Three Version Rule for Microsoft products. It was policy. XP changed all that, and made companies feel safe buying sooner. Vista nearly brought it back. Paradox seems to have embraced the model of software quality that Microsoft had prior to XP, because their products seem to be in the same pre-release or not ready state that those early Microsoft products were.
The recent release of Hearts of Iron III is an example of that. This product had a scheduled release date of August 7th, 2009, and it did release. The game is generally unplayable by many users, myself included. In fact, it seems the game suffer worse on newer multi-core machines than on older single core systems. One of the most popular suggestions to make the game somewhat tolerable is to set the affinity of the process to a single core, disable the system page file, close other running applications, and minimize the game graphics and music. In other words, change the environment the game needs to run in, because the game cannot adapt correctly. Many of the issues people are complaining about are broken basic features and intolerable performance. Both of these issues should have been caught in testing, but the game shipped anyway. While I have every confidence that Paradox will fix the issues, why do they use their customers as beta testers? The answer that comes most often – ‘all the other game companies do it’. That is when I stated thinking about this issue.
I started to think about all of the software products I use, and their quality. It is true. Every popular game that I have bought in the last few years that had a fixed release date, has had quality issues. The forums for these games are filled with tales of horrible experiences for the users. It reminded me of the discussions in the pre-XP era of Microsoft. I remember from back in those days having a product manager at Microsoft tell me that hitting the ship date is the only criteria for success. Someone I had worked with over the years used to tell me that ‘we can always change the definition of success.’ I am starting to believe that game companies are using those same maxims to determine success for their own products.