« News
Jul
25
On-going server development.
We've had a long week of very carefully moving database tables from one place to another, and then fixing all the related code. This is pretty delicate, time-consuming and tedious work, but it's critical to the upcoming "server migration" that has been referenced in the last couple of update notices.
For many years, we've run almost entirely out of a single datacenter, operating from hardware we own. But times have changed, we need to be able to scale in the face of possible large player counts from feature-promotions on iPhone, Android, Steam, and the like. "Cloud" computing services allow a degree of elasticity that will strongly benefit the game, letting us spin up a large amount of hardware when briefly needed, and then turn it back off again when the demand goes down.
For instance, this enables options like firing up a huge dedicated server to handle a massive sector battle-event for a few hours, and then shutting it down when the event wraps. The cost of creating that "new" server for a couple of hours would only be a few cents, compared to the big cost of our dropping several grand on a big server (and needing to host it, maintain it, power it, cool it, and having it sit idle when not-needed). By the same token, this same scalable infrastructure can potentially help us absorb hundreds of thousands of new iPhone players for a week, or other difficult-to-predict spikes in player concurrency.
The game architecture itself was always built around this model, even going back to the 90s, as we always wanted to be able to scale the game very quickly. So that part is pretty easy. Some players may even recall that we ran the whole game off of a cloud provider for a week, as a testbed experiment, before pulling it back to our own (less costly) hardware.
But, even with that, there's still some legacy development-baggage from having all your hardware in one place. Right now we're primarily focusing on changing the way the website and the game will communicate, to allow us greater freedom to host the game itself (or parts of it) anywhere, while the website and secure billing data will remain on physical hardware we actually own. This means the connections and services must be robust to potential disconnects from fiber cuts, routing flaps and other unexpected internet outages.
In the meantime, of course, we're also doing a lot of administrative "cleanup", updating and further hardening the security of various systems, while making others more flexible and robust. We originally budgeted a week for this process, but we keep discovering legacy stuff we need to take into consideration with the new design. It's amazing the amount of code cruft and legacy services you end up with, after 13+ years of running an evolving, production MMORPG.
Given that context, a couple of weeks of cleanup is probably not surprising? Still, we're hoping this will mostly wrap up next week, in terms of what is soaking up most of our development time.
After that, we'll be back to patches and release updates, and in a much better place to continue the push for VO 1.9 and beyond!
For many years, we've run almost entirely out of a single datacenter, operating from hardware we own. But times have changed, we need to be able to scale in the face of possible large player counts from feature-promotions on iPhone, Android, Steam, and the like. "Cloud" computing services allow a degree of elasticity that will strongly benefit the game, letting us spin up a large amount of hardware when briefly needed, and then turn it back off again when the demand goes down.
For instance, this enables options like firing up a huge dedicated server to handle a massive sector battle-event for a few hours, and then shutting it down when the event wraps. The cost of creating that "new" server for a couple of hours would only be a few cents, compared to the big cost of our dropping several grand on a big server (and needing to host it, maintain it, power it, cool it, and having it sit idle when not-needed). By the same token, this same scalable infrastructure can potentially help us absorb hundreds of thousands of new iPhone players for a week, or other difficult-to-predict spikes in player concurrency.
The game architecture itself was always built around this model, even going back to the 90s, as we always wanted to be able to scale the game very quickly. So that part is pretty easy. Some players may even recall that we ran the whole game off of a cloud provider for a week, as a testbed experiment, before pulling it back to our own (less costly) hardware.
But, even with that, there's still some legacy development-baggage from having all your hardware in one place. Right now we're primarily focusing on changing the way the website and the game will communicate, to allow us greater freedom to host the game itself (or parts of it) anywhere, while the website and secure billing data will remain on physical hardware we actually own. This means the connections and services must be robust to potential disconnects from fiber cuts, routing flaps and other unexpected internet outages.
In the meantime, of course, we're also doing a lot of administrative "cleanup", updating and further hardening the security of various systems, while making others more flexible and robust. We originally budgeted a week for this process, but we keep discovering legacy stuff we need to take into consideration with the new design. It's amazing the amount of code cruft and legacy services you end up with, after 13+ years of running an evolving, production MMORPG.
Given that context, a couple of weeks of cleanup is probably not surprising? Still, we're hoping this will mostly wrap up next week, in terms of what is soaking up most of our development time.
After that, we'll be back to patches and release updates, and in a much better place to continue the push for VO 1.9 and beyond!