Forums » Bugs
Server outage
We're currently looking into the issue.
The game server experienced an unplanned outage, due to a code bug, at 12:57 CDT, and was back online by 01:08 CDT (~11 minutes down).
We've determined the general cause, and come up with a workaround that will prevent the same issue from happening again, while we continue to investigate "why" it happened.
For a little background, the last month has had a lot of intensive server-side development; probably more than in the previous 20 years. So, with "new code" comes the potential for "new bugs".
I'll leave this thread here for awhile, and then eventually move it to Bugs.
We've determined the general cause, and come up with a workaround that will prevent the same issue from happening again, while we continue to investigate "why" it happened.
For a little background, the last month has had a lot of intensive server-side development; probably more than in the previous 20 years. So, with "new code" comes the potential for "new bugs".
I'll leave this thread here for awhile, and then eventually move it to Bugs.
Am I the only person that almost peed with excitement when I read this?
"For a little background, the last month has had a lot of intensive server-side development; probably more than in the previous 20 years. So, with "new code" comes the potential for "new bugs"."
"For a little background, the last month has had a lot of intensive server-side development; probably more than in the previous 20 years. So, with "new code" comes the potential for "new bugs"."
Yes. Many of us are not incontinent.
But I agree that the idea of a lot of server-side changes may mean cool stuff not far away. Incarnate will let us know at some point.
But I agree that the idea of a lot of server-side changes may mean cool stuff not far away. Incarnate will let us know at some point.
Okay, well, just to tie this off.. the originating core bug has been found and fixed, along with the more "general solution" to prevent similar issues down the road. Both have been pushed into production.
As it turned out, the problem was not related to the more fundamental "re-writing the core server" stuff we've been doing, but rather a kind of simpler tweak to give better alarms and handling of occasional "stuck sector" issues (stuck-sectors are not always caused by situations we can prevent, sometimes they're provider-related).
One of the effects from this "tweak" is that, rather than being able to jump into a "stuck" sector (and then get stuck), users may instead be sent back to their previous sector, if a problem is detected in the next-sector. We had some issues with incorrect problem-detection last week, but that should now be improved.
So, anyway, the crash-related bugs have been resolved and those specific issues shouldn't crop up again.
But, we are still doing a bunch of work in the most basic, low-level elements of the core game server, so future instability remains more likely than usual.
As always, your continued patience is appreciated.
As it turned out, the problem was not related to the more fundamental "re-writing the core server" stuff we've been doing, but rather a kind of simpler tweak to give better alarms and handling of occasional "stuck sector" issues (stuck-sectors are not always caused by situations we can prevent, sometimes they're provider-related).
One of the effects from this "tweak" is that, rather than being able to jump into a "stuck" sector (and then get stuck), users may instead be sent back to their previous sector, if a problem is detected in the next-sector. We had some issues with incorrect problem-detection last week, but that should now be improved.
So, anyway, the crash-related bugs have been resolved and those specific issues shouldn't crop up again.
But, we are still doing a bunch of work in the most basic, low-level elements of the core game server, so future instability remains more likely than usual.
As always, your continued patience is appreciated.