Suggestion: don't take the server down for maintenance during a 7 player conq station battle! Are you friggin' serious? lol...while you're at it please unbug my Trident. It glitched out and shows in sector but it's not actually there.

Found you

There's no news or forum post or anything. It's possible they didn't plan on this being an outage.

Devs should change the default login failed response then, because the current one deceptively states it's down for maintenance and will return shortly.

Your trident broke the game and now you wanna complain?!?

QQqqQqQQ

But seriously, what the feck is going on devs?

Wednesday morning is obviously not a planned outage, I would think...

Going on 90 minute outage now. I submitted a support ticket an hour ago. Maybe the devs are all having brunch at iHop or something. Sipping overpriced coffee and eating pancakes made by that ho that works with SynMat.

Westactical? Seven player conq station battle? Where the f*ck? Oh yeah, devs please please stop, these unscheduled outages are outrages.

It was a rockin 7 man conq station battle, lots of lulz were had until suddenly...!

FLAGRANT SYSTEM ERROR
VENDETTA OVER
STATION = VERY CRASH

Believe it or not I've seen some log files that were less descriptive than that

I believe it. I see them all the time.

Wow guys. Calm down. Stuff breaks sometimes. The devs are probably too busy panicking and trying to fix it to spend any time burbling comforting words in your ears.

Excessive alts cause server failures

This was not a planned outage. Also, for some reason, one of Ray's newer automatic monitoring scripts was not running (we're still cleaning up the server move, even after all this time).

We sometimes end up working pretty late (especially if we're trying to finish something), which makes us a little less on the ball in the morning, and we usually rely on automated monitoring to report these kinds of issues.

It was a weird issue to monitor as well, as only one subsystem (Erlang) freaked out, rapidly allocated gigs of memory until it ran out and crashed, wrote out an 824MB dump file for 20 minutes, and then restarted itself, which left things in a "maintenance" mode, basically. So our classical monitoring of fundamental server processes (not-Erlang) showed no issues.

We wrote a new monitoring system this summer to help with that, which actually does periodic headless-client logins to check game state, but like I said.. it wasn't running for some other reason.. which we're also looking into.

We may need monitoring to monitor our monitoring. And also monitor that. Sigh.

Why use Erlang at all? Isn't that something left behind from ages ago? Can that not be updated to something else? I mean it is just another scripting language right? So you have C, C++ with Lua and Erlang (and broken parts like debilitator and stuff).

Why not just use 1 or the other?

[edit]

Oh yeah sorry. You would need more than one programmer for that to happen.

Er, no? Erlang is actually pretty state of the art. We can simulate and track millions of characters in a gig of ram. There are very few other runtimes in existence that have that kind of lightweight threading. Plus we can dynamically spread the Erlang environment over multiple machines, at will.

The runtime environment is not the problem, our code is the problem. We just not sure how, yet. Bugs happen, no matter what language you use..

[edit] Ok, you edited you question. No, we would use C++, Lua and Erlang no matter what, or several different languages. They serve different purposes, and have different values. For instance, we often do server-side changes that require zero restarts, because they're all in lua and are just propagated out organically. Erlang is also created as a "hot-loadable" system. C++ is not, but it's really fast and uses little memory, so it's used for "heavy lifting".

That's basically a much larger discussion. And it has nothing whatsoever to do with the number of people we have.

[edit2] I think you're confusing a number of different points from history. We had a Lisp runtime, once, called "Deliverator", starting in mid-2005. It was replaced around February of 2006 with Kourier, which was re-written in Erlang. We've been using that ever since, and stopped using Lisp. "Deliverator" does not exist anymore, and hasn't since we moved to Erlang. Erlang has been pretty solid for us, actually. The fact that the Erlang codebase was a problem here has no bearing on anything.. the game server is a complex, distributed environment. We could write everything in C, everywhere, and still have a single subsystem failure. It's just as easy to have memory leaks in C. We just cleaned a few up in the client a month or so ago (very slow leaks).

It may seem counter-intuitive, but if we wrote everything in C, the game would be slower to develop and harder to maintain.

As a fellow programmer that makes perfect sense. Different tools have different uses, and while these languages are all Turing complete, it would be silly to replace the dynamic scripting parts in C and AFAIK erlang is the go-to language for real-time concurrent systems.

(BTW it's cool that you guys are listed on wiki under "projects using Erlang" next to i.a. github, couchDB, whatsapp and Goldman...)