NeverSawUs

Javascript

The Magic Parts

If there were a real world equivalent of the "Spinning Newspaper with Headline" trope so often seen in films, Javascript's meteoric rise to fame would involve headlines like the following:

  1. Javascript! Not Just for Browsers Anymore!
  2. After Reading Crockford's "The Good Parts", the President Resigns and Installs Node.js as Interim President
  3. JS Is Callback Soup, Declares Conservative Pundit
  4. Is the Honeymoon Over? Javascript's Uncanny Resemblence to the Elephant Man

Which is to say, there's a lot of hyperbole on both sides. What irks me in particular is that on both sides, there's a fair amount of misunderstanding of the language in question. On the one hand, there are those who just read Crockford's seminal work, "Javascript: The Good Parts"; they can usually be identified by their unbound enthusiasm for Javascript and how it implements first-class functions and closures. A key phrase to look for is something along the lines of

It's like LISP in C's clothing!

Which, I'm fairly sure, misses the point entirely — You can do closures in Python and Ruby, and hell, even PHP has them now, and quite frankly, they're implemented in a way that is a little bit more sane than Javascript's.

On the other side, you have folk deriding javascript for encouraging callback soup, and being an ugly language in general. A key phrase to look for, aside from the obvious "callback soup", is

I just don't get the excitement about Node.js.

Again, "callback soup" isn't something that just happens when you program in Javascript. If your program is well-structured, there's a lot of ways around it — it just requires one to learn some new tricks — and what is programming but an accumulated knowledge of different tricks (or, ahem, "patterns", if you will) that apply in different situations? I'll warrant, with the advent of DOM-manipulation frameworks like jQuery, callback soup has quickly become the norm, but that's not Javascript's fault.

As for the ugly language argument — yes, it's uglier than Python, and maybe just a little prettier than PHP. There are ways around this. But it's really not that bad — and if you think of Javascript as a superset or predecessor to Python or Ruby, it starts to make sense. Even if, chronologically, this isn't really the case, it does make sense; Javascript has all of the nice array and string manipulation features of the Big Two dynamic languages, just in a syntax that has a little more in common with the 90's than most programmers would like.

I'm going to digress for just a bit here, because I think at the root of both arguments is a misunderstanding of the "Magic Parts" of Javascript. In my estimation, there are two biggies, and it's really hard to wrap your head around them — until you get them, Javascript will be a source of consternation. So without further ado:

The First Magic Part

What is this?

Closures are super great awesome pieces of functionality that prove Javascript's worth as a language, right? Javascript is totally unique in that it supports them, yeah? No. Javascript is not special because it has closures, or first class functions, or whatever. 

# python
def needs_auth(fn):
    def wrapped(request, *args, **kwargs):
        if request.user.is_authenticated():
            return fn(request, *args, **kwargs)
        else:
            return redirect('/login/')
    return wrapped

# php
function needs_auth($fn) {
    return function() use ($fn) {
        $args = func_get_args();
        $request = $args[0];
        if($request->user->is_authenticated()) {
            return call_user_func_array($fn, $args);
        } else {
            return redirect('/login');
        }
    };
};

Python and PHP have closures and inline function definitions too. They are nothing special at this point. That might be a good rubric for dynamic language features — if PHP supports a feature, it is no longer the exception, it is an expected standard feature. So closures aren't magic. However, what they do and more importantly, don't close over in Javascript is magic. Another example in python: 

class A(object):
    def return_callback(self, *args):
        def cb(*nargs):
            return (self, args, nargs)
        return cb

cb closes over self, and whenever that callback is run, an instance of A is returned along with the other arguments. Let's look at somewhat equivalent Javascript code: 

var A = function() { 
    return {
        returnCallback:function() {
            return function() {
                return [this, arguments];
            };
        }
    };
};

Fun fact! This doesn't work. Why? Because whenever you see a function inside of a function, you cannot assume that this inside that function is the same as the containing function. This throws many people (myself included) for a loop when programming Javascript. Functions in Javascript are never strongly bound to an instance of an object, unless you explicitly make them bound using one of the following techniques: 

var returnCallback = function() {
    var self = this;    // <--  since self is a normal non-magic variable, functions created
                        //      within this scope will close over it.
    return function() {
        return [self, arguments];
    };
}; 

var a = {
    doSomething:returnCallback
};

// method ONE
a.doSomething()             // calls returnCallback with this set to a.

// method TWO
returnCallback.apply(a);    // calls returnCallback with this set to a.

// super secret method THREE
var Class = function() {
};
Class.prototype.doSomething = returnCallback;

var b = new Class();
b.doSomething();            // calls returnCallback with this set to b.

Any one of these three methods of binding a function to an object instance will cause this to be set.

That's the first magic property of Javascript — this is a magic variable, and will not be closed over independently. At first this is a major tripping point, but eventually you get used to it — whenever you see function inside of another function, if you need this, you will have to close over self instead.

Coffeescript makes this easy: 

returnCallback: () ->
    return () =>
        this

So if you want to pave over the magic, you can — with the => operator, which binds the resulting function tightly to whatever the value of this is at the function definition.

The Second Magic Part

Prototype is weird

Object.prototype. The mere mention of prototype sends many into a frothing fit. I suspect, personally, this is because there are so many different ways of defining classes in Javascript — "The Good Parts" exacerbates this by presenting two or three different ways in the chapter about defining classes. In general, none are particularly endorsed over another, and so there is much confusion about how to define a class (and don't even start to ponder how to define a subclass).

The basics are easy to grasp, however — let's take a look at some Python code. 

class Actor(object):
    message = "Pontificate!"
    def say_something(self):
        return self.message

gary_busey = Actor()
print gary_busey.say_something()    # <-- prints "Pontificate!"

This is fairly easy to follow. Gary Busey is an Actor, and when he says something, he looks up his message property and returns it.

But what's really going on is a little bit more complicated, and actually ends up looking a lot like how Javascript's prototypes work. 

gary_busey2 = Actor()
gary_busey3 = Actor()
gary_busey3.message = "TEEEETH"
Actor.message = "Act a fool"

gary_busey.say_something()  # "Act a fool"
gary_busey2.say_something() # "Act a fool"
gary_busey3.say_something() # "TEEEETH"

What happened?! Well, it turns out that when message is not defined on the object instance itself — which, for buseys 1 and 2 is true -- it checks the object's class for that property. By altering the class-level message, we alter what any actor without a message property says by default. You can override this by setting message on an Actor instance, and when it tries to look up that property, it'll find the local message first.

So in Javascript, objects work in almost exactly the same way! 

var Actor = function() {
};

Actor.prototype.saySomething = function() {
    return this.message;
};

Actor.prototype.message = 'Pontificate!';

var busey1 = new Actor(),
    busey2 = new Actor();

busey2.message = 'TEETH';
busey1.saySomething()       // 'Pontificate!'
busey2.saySomething()       // 'TEETH'

Actor.prototype.message = 'Act a fool';
busey1.saySomething()       // 'Act a fool'
busey2.saySomething()       // 'TEETH'

So it works the same way as Python in this case. It's important to note that when an object instance looks up a property on its prototype, it pretends that property was found on the object instance itself — hence saySomething's this is set to whatever actor instance invoked that method.

Prototypes can be chained, as well, to provide subclassing. There's another piece of weirdness here, too — to get prototype chaining to work, the object you are trying to subclass must be instantiated so that it has its prototype set. Oof — that's a mouthful. Let's take a look: 

var a = new Actor();    // returns an Object with its prototype and constructor properties set.

var BadActor = function() {
    Actor.call(this);   // we have to manually invoke the `Actor` constructor here.
};
BadActor.prototype = new Actor();       // this returns an Object with its prototype and constructor properties set
                                        // note that this is not the same Object as `Actor.prototype`.
                                        // it's a new object, whose `prototype` is set to `Actor.prototype`.
                                        // Any changes to `BadActor.prototype` will not propagate back up.

But wait that sucks. If there's any sort of complex logic in the Actor constructor, it's getting uselessly called when we define the BadActor.prototype. This is silly. To solve this, we use intermediary functions: 

var F = function() {
};
F.prototype = Actor.prototype;

var BadActor = function() {
    Actor.call(this);       // we still have to manually invoke the constructor
                            // but hey, that's not a lot different from Python's
                            // super(BadActor, self).__init__()
};
BadActor.prototype = new F();

var busey = new BadActor();
busey instanceof Actor;     // true
busey instanceof BadActor;  // true

var reeves = new Actor();
reeves instanceof Actor;    // true
reeves instanceof BadActor; // false

So that was a bit of a roundabout journey, but the essence is this: We delegate to a helper constructor who has the same prototype as Actor to create our BadActor prototype appropriately. instanceof tests to see if any prototype object in the chain of prototypes available on the provided instance match the prototype property of the provided constructor.

This can be a little mind-bending, but in practice, it ends up doing exactly what Python would do for an inheritance chain. This bears playing with — here's where I recommend you install node.js and use the command line REPL to run some tests for yourself until you feel comfortable with how prototype works.

Wield the Magic Parts like a Wizard

Mess up the Baelrog of code soup something fierce

One thing that is absolutely key to working in Javascript is understanding this philosophy:

If a function blocks in any way — by spinning a hard drive, by making a request, by pausing to observe a butterfly, it has to take a callback.

There should be no transparently blocking function calls. If it blocks, it takes a callback detailing what should be done when that function receives data. This is a big source of where "callback soup" comes from — the fear that your humble method will have to provide callbacks two or more levels deep to get any meaningful work done. This can be avoided by abstracting the callbacks away through proper framework design. By understanding and using the two magic parts above, it becomes much, much easier to structure your Javascript program to avoid callback soup.

An aside: I've noted that for many, writing Javascript is tantamount to writing jQuery. While I think jQuery is best-in-class at what it sets out to do — DOM Manipulation — it doesn't really teach you any Javascript. It feels a lot like PHP — a lot of the harder parts of Javascript are abstracted away and made easy to deal with, at the expense of not knowing what's really going on. It encourages you to juggle chainsaws by telling you that they are bowling pins. Without knowing how Javascript works, it's hard to structure your program in such a way that doesn't fall into the trap of callback soup — and harder yet to avoid depending too much on external DOM state. Code written for jQuery should not be taken to represent Javascript at large.

Abstracting Away Callback Soup

Just as a quick postscript — here is one potential way to abstract away callback soup. Our goal is to process a request/response cycle without requiring more than one callback per middleware function. Without further ado, the Request object: 

var Request = function(req, resp, callbacks) {
    var requestProcessors = callbacks.slice(),
        responseProcessors = callbacks.slice().reverse(),
        exceptionProcessors = callbacks.slice().reverse();

    this.req = req;
    this.resp = resp;
    this.continue = this.continueRequest;
};

Request.prototype.continueRequest = function() {
    try {
        var mw = this.requestProcessors.shift();
        if(mw) {
            if(this.mw.processRequest) {
                this.mw.processRequest(this);
            } else {
                this.continue();
            }
        } else {
            this.respond({
                status_code:404,
                headers:{},
                content:'',
            });
        }
    } except(err) {
        this.error = err;
        this.continue = this.continueException;
        this.continue();
    }
};

Request.prototype.continueResponse = function() {
    try {
        var mw = this.responseProcessors.shift();
        if(mw) {
            if(this.mw.processResponse) {
                this.mw.processResponse(this);
            } else {
                this.continue();
            }
        } else {
            this.respond(this.response);
        }
    } except(err) {
        this.error = err;
        this.continue = this.continueException;
        this.continue();
    }
};

Request.prototype.continueException = function() {
    try {
        var mw = this.exceptionProcessors.shift();
        if(mw) {
            if(this.mw.processException) {
                this.mw.processException(this);
            } else {
                this.continue();
            }
        } else {
            this.respond({
                status_code:500,
                headers:{},
                content:'',
            });
        }
    } except(err) {
        this.error = err;
        this.continue = this.continueException;
        this.continue();
    }
};

Request.prototype.respond = function(response) {
    if(this.continue === this.continueRequest) {
        this.response = response;
        this.continue = this.continueResponse;
        this.continue();
    } else {
        this.end(response);
    }
};

Request.prototype.end = function(response) {
    this.resp.writeHead(response.status, response.headers);
    this.resp.write(response.content);
    this.resp.end();
};

So what does this do? Well — we want to iterate down a set of potentially blocking middleware until we get a response, and then iterate back up that middleware allowing it to modify said response. If it doesn't encounter a response, it 404's, and if it catches an exception, it iterates down the potentially blocking processException middleware.

So middleware can look like this: 

var LOLMiddleware = {
    'processRequest':function(request) {
        if(request.req.headers['Cookie']) {
            db.getSession(request.req.headers['Cookie'], function(session) {
                request.session = session;
                request.continue();
            });
        } else {
            request.continue();
        }
    }
};

As you can see — we've got one callback in this code. It's just asking the database if there's a session by the name of this cookie. If there is no cookie set, we just continue the request. The fact that we're iterating down a series of callbacks is almost completely transparent. The request.continue call is present due to the inversion of control that happens when callbacks are used — but it's not too specious, it merely signals that this stage of processing is done and that the request should continue on to the next potentially blocking operation.

Callback soup is what happens when there isn't a well-thought out structure for dealing with callbacks. It is not implied by using Javascript, rather, it happens when Javascript is used thoughtlessly.

You Just Graduated from Hogwarts.js

Pick up your wizard robe on the way out

Javascript isn't the end all and be all because it has first class functions — it's exciting because it fits the event-based model of programming very well.

It's not exciting because it's portable — the fact that you can run the same code on the server as on the client is, at best, glossing over the truth of the matter; at worst it's an outright lie. Javascript is a different language on IE, Firefox, and Webkit. It's not the same: you have to program in a very strictly limited subset of Javascript that is known to work on all three. What's exciting is that there are three major browser vendors with a vested interest in improving the performance of Javascript on their respective platforms. There's a lot of mindshare in Javascript right now — arguably more than any other dynamic language at this point. That's why Javascript is exciting.

It's not exciting because of "The Good Parts" — and it's certainly not because it's "write once, run anywhere". It's the mindshare, it's the succinct way in which it fits the paradigm of event based programming. It's fun to use because it's relatively green field, and there are many important libraries that need to be written — and it presents a new set of problems in framework/library design as compared to Ruby and Python.

That's the real reason to get excited about Node. It's fun to learn new patterns, it's fun to write code in when you get the two magic parts of it, and it's not entirely unlike programming in any other dynamic language.

And it's only going to get better as more people write code and experiment with it.