deflect.jsAn ECMAScript module fusing Node.js error-first convention with continuation passing style, plus a twist to make composed stacks of asynchronous functions dynamic
I recently released an npm package, Deflect, for use in a personal API server project, but also as a thought experiment in combining error-first callbacks, continuation passing style, and a twist on continuation passing that I haven’t seen before.
Deflect combines:
- Composition of stacks of potentially asynchronous continuation passing functions to be applied successively to arguments, like Async’s
seq. - Shoehorning of thrown errors into the error-first paradigm, like Connect.
- Dynamic, per-invocation mutation of the function stack by functions within the stack.
- Callbacks that throw errors on double invocation, like Once.
The package’s sole export is a higher-order utility function for composing such functions, and even though a practical looking example is part of the test suite, I felt I’d like to write out a broader statement of the what and why. Better here than weighing down the code repository.
Motivation
ECMA Script programmers have developed several strong conventions not enshrined in the language itself, including error-first callbacks and the use of continuation passing style for asynchronous operations.
Both conventions can now be found in core Node.js APIs. For instance, in fs:
var fs = require('fs');
fs.readFile(
'/etc/passwd',
// This is the "callback function."
// Notice that the first argument is an error.
function(error, data) {
// If readFile produces any kind of error, it
// will be passed as the error argument.
if (error) {
throw error;
}
// In the normal, desireable case, error
// will be falsy (probably null or undefined).
console.log(data);
}
);
Node core also evidences some continuation passing style, at least in relatively recent stream:
var stream = require('stream');
var t = new stream.Transform();
t._transform = function(data, encoding, callback) {
// Perform the desired, potentially asynchronous operations.
callback();
};
This usage is somewhat atypical, in that the continuation passing function is not an error-first function. That combination is omnipresent outside Node core, often facilitated by the very popular async npm package:
var async = require('async');
async.series(
[
// Continuation passing style
function(next) {
next(new Error());
},
function(next) {
next(null, true);
}
],
// Error-first callback
function(error, result) {
// ...
}
);
The majority of Async’s higher-order functions take a second error-first callback argument, which is invoked immediately if any composed continuation passing function invokes its callback with an error. In this way, the functions that attempt to produce desired results are stacked separately from those that handle errors. Though there is nothing to prevent Async being used to compose error handling functions, it is primarily billed for composition of asynchronous and error-prone result producing functions.
The same distinction between main-line logic and error handling functions is seen in Connect, Node’s original analog to Ruby’s Rack. (Connect was once used by Express, the current king of minimal Node HTTP server frameworks, as Ruby’s Sinatra utilized Rack.) Connect, at least as of version 3, is implemented as a function whose prototype stores an stack of handler functions that are invoked in order when the containing function is called.
var connect = require('connect');
// By default, the stack of functions contains only a
// default final handler that throws an error.
var app = connect();
// Push a handler function onto the stack.
app.use(function(request, response, next) {
// Write to response, or defer to remaining handler
// functions on the stack by calling next.
});
// Push error handling middleware onto the stack.
app.use(function(error, request, response, next) {
// Handle the error, or defer to the next error handling
// middleware.
});
Connect’s API exposes one function, use, for pushing functions onto its internal stack, and distinguishes internally between handler functions (arity 3: request, response, and callback) and error handling functions (arity 4, with an additional error-first argument) by use of .length. Error handling functions are invoked when a handler function yields next(error), but also by catching thrown errors and passing them on as error-first arguments. Throw-try-catch flow control is shoehorned into the error-first paradigm.
In addition to a potential error, Connect also passes the Node core http.IncomingMessage (request) and http.ServerResponse (response) arguments from the http.createServer handler function.
Unlike the error argument, which may or may not be null or passed down the function stack, request is passed to each function by Connect itself. While each function will receive a pointer to the same request object, that object need not be mutated to direct a valid HTTP server response. As a practical matter, however, a great many middleware add properties to or otherwise mutate the IncomingMessage object, such as by replacing a serialized JSON request body with a parsed Object, by stripping extraneous trailing slashes from request paths, &c. It is also a common pattern to graft a pointer to a database client or abstraction layer to a property of request, for convenience and want of a better place for it.
The request argument is also passed to successive handler functions by Connect, but per the HTTP API that object must be mutated per-request (via prototype function calls) to indicate how the core http server logic should respond. Middleware can mutate the response object and then call back to successive middleware, creating the need for awkward APIs and the potential for errors on, say, attempts to write additional information when packets have already been sent.
Connect’s automatic passing of the request and response objects from the underlying HTTP API specializes it for HTTP server creation. It also forces middleware authors to embrace mutation of the request or response objects to make information available across middleware without closing over variables polluting global or some more limited, but nonetheless shared, scope.
Existing tools
When viewed as higher-order functions that compose stacks of potentially asynchronous functions applied in sequence, Connect and async.seq are of a kind. Internally, both maintain a stack of functions whose elements are successively applied to the results of previous applications or initial input. (Connect also has prefix-based path routing built in, ignored here.) We could in fact write an asynchronous http handler function with async and without Connect, structured perhaps a bit more like Rack:
var http = require('http');
var async = require('async');
// Factory for text/plain resource handler functions.
var textResource = function(path, text) {
return function(env, next) {
if (env.request.url === '/' + path) {
env.response.writeHead(200, {
'Content-Type': 'text/plain'
});
env.response.write(text);
}
next(null, env);
};
};
var app = async.seq(
// Middleware to trim trailing request path slashes.
function(env, next) {
var url = env.request.url;
if (url[url.length - 1] === '/') {
env.request.url = url.slice(0, url.length - 1);
}
next(null, env);
},
// Handler for the /a resource
textResource('a', 'A!'),
// Handler for the /b resource
textResource('b', 'B!'),
// Handler for /c, rigged to error out
function(env, next) {
if (env.request.url === '/c') {
next(new Error());
} else {
next(null, env);
}
}
);
var handler = function(request, response) {
// Package request and response in a single
// object for passing on down via .seq
var env = {
request: request,
response: response
};
// Invoke the function created by async.seq
app(env, function(error) {
// If there's an error, send an error
// status code.
if (error) {
response.statusCode = 500;
}
response.end();
});
};
var server = http.createServer(handler);
In this way we put the job of ensuring that relevant request and response pointers are passed from middleware to middleware in the hands of middleware itself, much as Rack, back in the Ruby world, initializes middleware objects with pointers to subsequent middleware that should be invoked with the env hash.
Another permutation
The new module, Deflect, fuses error-first callbacks and continuation passing functions into a single convention for functions that each take an error, an arbitrary list of additional arguments, and a continuation. Rather than require separate stacks of functions for result generation and error handling, each function is passed all results, error or otherwise, yielded by or caught from preceding functions.
Functions are composed as follows:
var fs = require('fs');
var deflect = require('deflect');
// Compose error-first, continuation passing functions.
var composed = deflect([
// Read a file.
function(error, fileName, next) {
fs.readFile(fileName, next);
},
// Print the length of the file's content.
function(error, content, next) {
console.log(content.length);
// Pass two additional (non-error, non-continuation)
// arguments to the next function in the stack.
next(error, content[0], content[1]);
},
// This function takes four arguments, including the two
// passed from the prior function.
function(error, first, second, next) {
var longer = first.length > second.length ?
first : second;
// It also yields a final value.
next(null, longer);
}
]);
composed('input.txt', function(error, result) {
if (error) {
// ...
} else {
// The result will be the longer of the first two lines
// of the file input.txt.
console.log('Final result is: ' + result);
}
});
Architecturally, we gain homogeneity among functions, which are now of a single, unified signature and convention. On the other hand, we lose specialization as we know it. In a composed stack of, say, seven functions, only the last of which handles errors, an error thrown by the first function will be passed through the succeeding five functions, which must if (error) { ... } or the like, before it reaches the seventh.
This would be a problem if, as with Async or Connect, we were limited to assembling one, static stack of functions that applies on each invocation. The continuation function passed to functions composed by Deflect facilitates a different approach: if a function or array of functions, rather than an error, are passed as first argument to a callback in the stack, the new functions are shifted to the front of an invocation-specific copy of the stack, to be executed next. The function stack, which is passed along as arguments to recursive function calls, can thus be self-modifying, with each invocation of the composed function starting afresh with the original stack of Deflect-composed functions. Functions of specialized purpose need not be applied needlessly on every invocation, but can be brought in by functions in the stack when appropriate.
To run with the theme of HTTP server handlers:
var deflect = require('deflect');
var requireAuth = function(error, request, response, next) {
// Parse the Authorization header, pull the user record
// from the database, check it against the header, and
// pass it along as an argument to the next function.
next(error, user, request, response);
};
var postXHandler = function(error, user, req, res, next) {
// Subsequent functions will not need the user object,
// so there is no need to yield it.
if (!user) {
// ...
next(error, req, res);
} else {
// ...
next(error, req, res);
}
};
deflect([
function(error, request, response, next) {
if (request.url === '/x' && request.method === 'POST') {
// With just a function argument, the callback will
// call the provided function with the arguments
// passed to this function.
next([
requireAuth,
postXHandler
]);
} else {
// Without arguments, the callback will reuse the
// arguments passed to this function with the next
// from the stack of composed functions.
next();
}
}
]);
Constraints
Use of Deflect is subject to two constraints.
All composed functions must accept, at a minimum, an optional error argument and a continuation, with additional arguments, if any, sandwiched between. The error argument to a function may be yielded from a previous function, but Deflect will also catch thrown errors and pass them as error arguments.
Functions in successive order, whether placed there by a call to deflect or as first arguments to a continuation callback, must yield and receive the same number of arguments. next(error, x, y, z) must be followed in the stack by a function of arity 5, function(error, x, y, z, next) { ... }. For visually oriented readers:
defect([
function ( error, next ) {
// ...
next ( error, true ); }, // 1 value
function ( error, x, next ) { // (same)
// ...
next ( error, x, true ); }, // 2 values
function ( error, x, y, next ) { // (same)
// ...
next ( error, x ); }, // 1 value
function ( error, next ) { // (same)
// ...
next ( error ); }
]);
In the future, Deflect may throw errors when the number of values yielded via a continuation does not match the non-zero .length of the next function on the internal stack. For now, deflect merely calls the next function blindly.
Thoughts and implications
Async combined the error-first and continuation passing conventions; Deflect tries fusing them together.
In some cases, Async’s separation of result producing and error handling logic may be more natural. That separation may reduce the need for boilerplate error checks in results-oriented middleware, though that kind of thing can be abstracted out fairly easily:
var unlessError = function(callback) {
return function() {
var error = arguments[0];
var next = arguments[arguments.length - 1];
if (error) {
callback.apply(this, arguments);
} else {
next();
}
};
};
deflect([
// ...
unlessError(function(error, input, next) {
}),
// ...
]);
The dynamic continuation passing aspect of Deflect strikes me as a potentially worthy substitute for duck-typed functions. But it also begs the question: Why use a funky looking library call to defer calling the next function in the stack instead of just invoking additional continuation passing functions with Deflect’s callback?
There is nothing to stop authors from doing just this, and I don’t imagine it will ever be appropriate to wrap each and every database query, crypto operation, or other non-blocking procedure in a Deflect-composable function. Deflect does provide some marginal benefits if you allow it to manage invocation, such as error catching and wrapped callbacks, but that could be done manually, and perhaps more judiciously or efficiently. It’s a pain to handle code in multiple conventions and paradigms, but that is the story of ECMA Script programming progress thus far.
The uniqueness I perceive in Deflect is that it is highly general and capable of unifying many counter-conventional control flow mechanisms (thrown errors, redundant invocation of callbacks, returning values without calling back) into the error-first continuation passing world. Mixing and matching those paradigms can be and is now a chore, but it is possible with Deflect because, like Async and Connect, Deflect takes it upon itself to manage a call stack within the program, rather than merely using the native, synchronous call stack mechanism provided by the language runtime. With the addition of next(function(...) { ... }), Deflect exposes a very limited degree of interactivity with the internal function stack to the library user, allowing them to bring conditional invocation and dynamic nesting within the library’s “managed” function stack mechanism. In that sense, it’s one step closer to an asynchronous eval(code, callback).
ECMA Script is a single-threaded, synchronous-by-default language privileged beyond its merit by tight integration with vital asynchronous APIs. With generator functions, futures/promises, and all manner of salvation on the offing, the willing pilgrim can take no shortage of enlightened paths full circle back to that fundamental dissonance. I have walked my own circle path with Deflect, and know better now how little I know.
If you’ve made it this far, I welcome any thoughts you may have. My contact information is available, as always, at kemitchell.com.
Your thoughts and feedback are always welcome by e-mail.