I know that it's been a little while since I've put up a post, but we've been heads down working on a number of important things that will help improve the game and online experience. Let me give you an update.
As many of you are painfully aware, we've had some problems (I'm risking understatement here) with the timely and accurate reporting of match stats. When these issues first arose, my first reaction was, "Yikes! We've got to fix this issue fast -- but at least it should be pretty straightforward." Boy, was I ever wrong. It turns out that the stats issues we've been seeing are not the result of a single bug but are several different problems that are all interrelated and only show up when the system is under extreme load. The good news is that most, although not all, of these problems have been resolved.
What's Been Fixed
• The backend stats processing infrastructure is quite sophisticated, but unfortunately there were a couple of components that did not scale as well as we had hoped. This led to a backlog of match results that, at times, could be as long as three or four days. While these results were never in danger of being outright lost, when you don't see the matches showing up in your stats for days on end, it's almost as bad as if they never showed up.
This problem has now been fixed and the system is now processing stats at a pretty good clip. The back log has been cleared so I'm afraid that if you are still missing stats they may fall into one of the categories described below and are probably lost. Our preliminary testing has shown that generally match results are being processed in close to real-time; most results are making it into the backend database in about 15-30 seconds, with the website usually receiving them within just a few minutes.
• There were a few website bugs that were preventing stats and match history from displaying correctly. One of these bugs caused the “match time” stat in the history list to be set to the time that the match was processed, and not the time that the match ended. When our backlog was a few days, this meant that matches in your history had timestamps with little relation to when the match actually occurred. This problem has been fixed and all new matches should be getting the correct timestamp.
• It doesn't’t have anything to do with stats, but it’s definitely worth mentioning. With the 1.04 patch, lobby stability has been much improved. We’ve opened up many more lobbies because the crux of the stability problem – crashes and disconnects -- was due to the strain of having four or five hundred people in a single lobby. We now load-balance people at login, and limit the maximum number of people in a single lobby to a little over a 100. Check out the new podcast, where the Producers talk about this issue in more detail. (BTW, when Amer talks about people on the Internet being crazy, of course he’s not talking about you.)
What's Still Out There
• There is still a known problem where a very small percentage of stats submissions – less than half of one percent – are being outright lost. We’re still investigating, but the fact that this so difficult to reproduce is definitely gating how quickly we can track it down. We can detect when it happens, so if nothing else we will probably make changes in the “Big Patch” that will give players some indication at the beginning of the match that the match stats will not be submitted. If you are still seeing a significant number of dropped stats with new games, and especially if your loss of stats has high reproducibility, it would be a big help if you could please respond to this thread and describe exactly what you are seeing.
• There are still a few people – at last count we’ve identified about ten people through forum posts – that have no stats history at all. They are definitely playing games, and even broadcasting those games fine, but not a single stat submission from them has come through. I know this is making some of you guys go, “WTF!” -- us too. We have already begun to contact these individuals and will be working with them to get this bug figured out. BTW, I don’ t mean to imply that only 10 people are being affected by this bug. There are certainly more folks who fall into this category but haven’t contacted us or haven’t posted on the forums. Still, it’s clearly a very small percentage of everyone playing, which makes tracking the bug down more difficult. Hang in there – we’ll get it.
• We, or in most cases our ISPs, have been hit by several denial- of-service hacker attacks over the past month. Unfortunately many (most?) stats submissions that occurred during this downtime are lost forever and we do not have any way to retrieve them. I’m afraid that when bandwidth gets saturated to the point where no network traffic can get to our servers, we obviously cannot receive and process stats submissions. Hopefully we’re out of the woods on this one.
• As noted in the podcast, we will be making some changes in the upcoming “Big Patch” that will return the themed lobby names, while retaining our load- balancing strategies and lobby population caps.
If you’ve listened to the latest podcast, you know that the team is hard at work on what we’ve been calling “ The Big Patch.” Unlike the little emergency patches that we released shortly after product launch, this one is the real McCoy and contains hundreds of bug fixes and minor new features (well, I know some of you wouldn’t call alt mouse setup minor… <g>). Amer and Greg K. talk about this in the podcast in some detail. Are you seeing the pattern here? Listen to the podcasts; there’s some valuable info to be found in there, amongst the cheese and jocular banter.
(As an aside, the second part of the podcast with my interview in it was recorded a week or so ago, so the information in this post supersedes what I may have said in the podcast.)
Once again, we really, really want to apologize to you if you’ve had a less than smooth experience with multiplayer. We are making good progress but as the above notes show, we've still got a ways to go. We will keep at it until it’s right. Mike and I, and the entire team, have been overwhelmed by the response that the game has been receiving. There are obviously many, many people having a great time with the game, which is the main reason why most of us have chosen this career. Thank you for the kind words and we look forward to lots of exciting C&C3 news in the weeks and months to come.
- Smudge
