Open Beta Launch Issues

The biggest issue we’ve had today is that getting into the Champions Online open beta was more popular than even we hoped for. While I know this doesn’t make the people who have been going through the challenges of getting online today any happier, it has been a great test for us in terms of diagnosing and working to fix problems on numerous fronts. In short, here’s what happened:

Our game and account servers were thoroughly tested with up to ~40,000 concurrent players. Unfortunately our game patching software was dependent on being able to ping the website to generate a dynamic information page, and it stopped updating when we experienced downtime from massive amounts of players hitting our web servers.

Concurrently, our installer had an error where it placed certain file types to an incorrect directory. When the patcher ran, it didn’t find those already downloaded files where they were expected to be. This caused the 3.6 GB patch to occur.

We quickly released an interim patcher that moved the already installed files to the correct folders. But as we had no way to message this through the patcher, this was incorrectly displaying as it being a 2 GB “patch.” It actually took only as long as moving the files, but unfortunately caused more confusion. The amount of data transferred by the patcher is the "Received" line.

There is now a new build of the installer at fileplanet.com that addresses the incorrect file placement and patch size, as well as a known Windows 64 Bit issue. We tried to keep the new install a simple single-file .exe autoextract, but those can't be bigger than 2GB in size. This requires players to do a second ~300MB patch after newly installing the game. If you previously downloaded the game from fileplanet.com, you do not need to do so again. The latest version of our patcher will move the incorrectly installed files and not require the previously massive patch.

Some players are currently experiencing issues where the Patcher can “freeze.” The main problem is we’re delivering a 400 MB patch to ~20,000 people simultaneously. At the current speed, the patching crush should clear up in about six hours. We are also currently working with our hosting partner to deploy more patch servers.

We appreciate the patience of our testers and hope that you’ll all stick in there as we go through our first day of getting the huge number of moving parts that is putting an MMO out there meshing together.