Daedalus' Forums

Hi guys,

Just letting you all know why it's been quiet around here, at least on my part.

A few months ago I started working for the sub-company at work, who specializes mainly in Search Engine Optimization (SEO).

The past few weeks has seen one of main projects come about, and to put it plainly, I've been working my butt off on it.
There are two reasons for that - I was given a short deadline to get it going, and then it was accidentally released to the public over a week early.

It wasn't meant to be announced until the end of the week, but we've had people using it already - that was a surprise to me, but a mistake thats proven the system works.

You may not be interested, but here it is anyway, and something I believe is the future of internet-based services: Cloud Computing.
I've written a system capable of servicing thousands of requests per second, sending those off to be processed by whatever server is available.

Now, this is not a new idea. Erlang was built on a very similar idea to what I've done, and they did it decades ago. However, look over the code - it's functional programming, and not easy to pick up.

Node
So instead, we're using parts of Erlang, and the rest is in a new programming language. Well, not quite new, but the usage is new.
We're using Node, which is a server-side version of Javascript. It's using Google's V8 Javascript engine, which compiles the code down to native code (not bytecode!)

Now don't laugh - Javascript is actually a powerful language, and the way it's being used in Node is closer to how Erlang and other functional programming languages in that code is a series of small, tightly contained function that will be called when the time is right.
The idea is to avoid code that blocks - that is, stops the code from running - instead you tell the code what to do, and give it a function to do when it has done what you requested.

This may not seem like much, but it means that while your program is waiting on a file to be read from disk, it's also processing an array of data, sending a request to a database server, and generating your response. As each piece of your puzzle falls into place, your callback for that particular task is called, and when everything is done you can finally give the user the data they wanted.
The upshot of this is that you can service multiple requests at once, without threading or having to deal with race conditions and such.

Cloud Computing
This brings us to the next part, and that which has had me working up to 10 hours a day the last few days.

Our main product already has over 130,000 paid users. The module I'm working on will soon be released to these users, and as you can imagine, an online service capable of serving this many people is not an easy undertaking.

It will have to stand up to hundreds, perhaps thousands of simultaneous requests at any given moment, and should be lightning quick to the end user.

What I've written, is a system that is separated into many parts, but all acting as a whole.
It is almost entirely decentralized, and we're able to spin up new servers at any time if the load starts becoming too much.

There is the frontend - the current single point of failure, one I hope to eradicate soon. Presently, each frontend is capable of serving over 500 requests per second, but this doesn't actually do any of the work. Its only job is to receive a request, and send it into the cloud; and to get any results of the request and give them to the user.

The cloud services are capable of running multiple jobs at once, and since Node is single threaded, we're running multiple instances of each service on each server. Each box isn't all that impressive - 256Mb servers running Linux, and I haven't actually checked their CPU power. But when you can start up 6 of these to process jobs, it amounts to a lot of processing power.

Finally, tying it all together, is a messaging system that is based on the OTP - the same system that powers the likes of Facebook Chat, for instance.
It is to here that user requests are sent, and the next available cloud server will be given the message, and can communicate back the response.

Using these simple but powerful pieces, I've crafted a system that is entirely unreliant on it's various parts - if something needs to be updated, we can direct the cloud services to another queueing server, or upgrade a service whilst it's running, and generally just make sure that no matter how hard one tries, they can't break the system.

Conclusion
So there you have it. You may not be excited about it, but I really am. I'm laying the groundwork for the future of my company, and quite literally doing stuff nobody else has done before - many of the libraries we're using have been written from scratch using whatever specs we could find, as Node itself doesn't yet have many libraries for what we're using.
Certainly nobody has done it with the tools we're using now, and I think the work I'm doing will have quite a lasting effect on my company as it moves forward.

Lastly, thank you to everyone for your patience and encouragement. The Revival Project is not dead, work is progressing, but at the same time I only have so much time I can devote to things.

That sounds pretty cool to me. There is an inherent elegance to the remote distributed processing/storage model, and I bet it is satisfying to implement.