Thread: Letter from the Colonel

Since the launch of America's Army 3 (AA3) on June 17, America's Army has seen data transfer demands upon our game infrastructure that far exceeded our expectations and capacity to support new game registrations and game play. Data transactions associated with all existing and new players attempting to complete the Basic Combat Training (BCT) portion of the AA3 game over the span of a few days overwhelmed our authentication and database systems. These sections of AA3 include key new features of the game that are particularly data intensive and which we clearly did not adequately load test during our closed and open beta process. As Director of the Army Game Project, I apologize for this gap in service and want to outline below the actions we have taken and are taking to close this gap.

First, AA3 is designed to provide players a great deal of data regarding their performance and key aspects of the online experience. As a result, compared with prior versions of America's Army, AA3 generates about ten times more traffic between game instances and our middleware and backend infrastructure. While we accounted for this level of data exchange when designing our AA3 infrastructure, we did not anticipate the unprecedented demand and data transfer levels we are witnessing as tens of thousands of players are simultaneously establishing accounts and working through the BCT portion of the Game. This was due to the fact that we did not adequately replicate the data transaction loads associated with this game activity during our beta test. As a result, immediately following the launch of AA3, we saw a mismatch between incoming data transactions and our infrastructure capacity. Therefore, we immediately began adjusting our infrastructure resources to maximize capacity and undertook substantial tuning of our database systems. Shortly after launch, we also discovered a hardware fault at our data center that created significant network lag. After addressing these faults, we achieved improvement in our infrastructure performance. However, demand for the game quickly consumed the capacity we created with the foregoing steps. Therefore, while we were able to support additional users, individual players did not see substantial improvement in game performance.

Since last Friday, to further improve system performance, we created a data handling modification for the game designed to moderate data flows during this launch period. This modification was released on Friday evening via Steam and our Deploy Client to automatically upgrade AA3 games installed on players’ computers. Additionally, we began crediting all players for completion of BCT levels so that they could proceed directly to multi-player missions; thereby removing the bottleneck created by the high volume of players engaging in BCT maps and improving the overall player experience.

We also began addressing performance and stability issues on the game servers by implementing memory upgrades and adding more database nodes to carry the load of authentication and player stats traffic between game instances and our infrastructure. We are correcting map rotation reporting deficiencies and issues with the Master Browser System (MBS) that prevent players from identifying servers with available Soldier positions. To improve network traffic, we continue to tune game network code, add more game servers, and correct server administration features so that server providers can better manage their servers. Finally, we are remedying instances where we identified memory leaks and are reducing server memory requirements to eliminate network lag while increasing server stability.

In summary, we are working to add capacity, and correct faults that have significantly impaired player access to America's Army 3. We appreciate player patience as we ramp up our infrastructure to meet the demand that clearly exceeded our capacity on launch day.

Casey Wardynski
Colonel, U.S. Army
Director, Army Game Project (America's Army)

From the AA forums
http://forum.americasarmy.com/viewtopic.php?t=300342