Real-time user count without any Javascript

Live Demonstration
(but see update 3 below)

IP cameras are frequently streamed as M-JPEG over HTTP connections. Why not use the same principle for other uses, too? Although M-JPEG streaming doesn’t work in Internet Explorer, it can still be useful.

The underlying protocol is simply a multipart HTTP request using the x-mixed-replace content type. Basically, the server sends the document (or in this case, image) multiple times, with each part replacing the previous part. So when an IP camera is streaming, it’s just sending multiple jpeg images that each replace the previous image.

It looks like this:


HTTP/1.1 200 Ok
Content-Type: multipart/x-mixed-replace; boundary=--icecream

--icecream
Content-Type: image/jpeg
Content-Length: [length]

[data]

--icecream
Content-Type: image/jpeg
Content-Length: [length]

[data]

--icecream

(and so on)

In this bare-bones example, the boundary is icecream, but ideally you’d want to use something that won’t appear in the data itself. Also, the content-type can be anything, such as image/png.

While learning about this, a practical application immediately came to mind: a real time online-users counter — such as the one above. This method is great for this use because it fills two roles. First, it keeps a persistent connection between the client and server, so we know when a user connects and when that user disconnects. Second, it can display the user count in real-time without requiring any page updates. This system can be used in places where Javascript is not an option, such as on a forum.

I made it using Node.js, with Backbone.js, and node-canvas — which unfortunately does not work on Windows. If you’re familiar with Backbone, the code should be rather self-explanatory.


(you can also download the source)

/* Real-Time PNG-Streaming HTTP User Counter
   Copyright Drew Gottlieb, 2012

   Free for any use, but don't claim
   that this is your work.

   Doesn't work on Windows because
   node-canvas only works on Linux and OSX. */

var moment = require('moment');
var http = require('http');
var _ = require('underscore');
var Backbone = require('backbone');
var Canvas = require('canvas');

var config = {
	port:	9192,
	host:	"0.0.0.0",
	updateInterval: 3000, // 5 seconds
	multipartBoundary: "whyhellothere"
};

var Client = Backbone.Model.extend({
	initialize: function() {
		var req = this.get('req');
		var res = this.get('res');

		console.log("Page opened:", req.headers.referer);

		res.on('close', _.bind(this.handleClose, this));
		req.on('close', _.bind(this.handleClose, this));
		this.sendInitialHeaders();
		this.set('updateinterval', setInterval(_.bind(this.sendUpdate, this), config.updateInterval));
	},

	// Re-send the image in case it needs to be re-rendered
	sendUpdate: function() {
		if (this.get('sending')) return;
		if (!this.get('imagecache')) return;

		this.sendFrame(this.get('imagecache'));
	},

	// Sends the actual HTTP headers
	sendInitialHeaders: function() {
		this.set('sending', true);

		var res = this.get('res');
		res.writeHead(200, {
			'Connection': 'Close',
			'Expires': '-1',
			'Last-Modified': moment().utc().format("ddd, DD MMM YYYY HH:mm:ss") + ' GMT',
			'Cache-Control': 'no-store, no-cache, must-revalidate, max-age=0, post-check=0, pre-check=0, false',
			'Pragma': 'no-cache',
			'Content-Type': 'multipart/x-mixed-replace; boundary=--' + config.multipartBoundary
		});
		res.write("--" + config.multipartBoundary + "\r\n");

		this.set('sending', false);
	},

	// Sends an image frame, followed by an empty part to flush the image through
	sendFrame: function(image) {
		this.set('sending', true);
		this.set('imagecache', image);

		var res = this.get('res');

		res.write("Content-Type: image/png\r\n");
		res.write("Content-Length: " + image.length + "\r\n");
		res.write("\r\n");
		res.write(image);

		res.write("--" + config.multipartBoundary + "\r\n");
		res.write("\r\n");
		res.write("--" + config.multipartBoundary + "\r\n");

		this.set('sending', false);
	},

	// Handle a disconnect
	handleClose: function() {
		if (this.get('closed')) return;
		this.set('closed', true);

		console.log("Page closed:", this.get('req').headers.referer);
		this.collection.remove(this);
		clearInterval(this.get('updateinterval'));
	}
});

var Clients = Backbone.Collection.extend({
	model: Client,

	initialize: function() {
		this.on("add", this.countUpdated, this);
		this.on("remove", this.countUpdated, this);
	},

	// Handle the client count changing
	countUpdated: function() {
		var image = this.generateUserCountImage(this.size());

		this.each(function(client) {
			client.sendFrame(image);
		});

		console.log("Connections:", this.size());
	},

	// Generate a new image
	generateUserCountImage: function(count) {
		var canvas = new Canvas(200, 30);
		var ctx = canvas.getContext('2d');

		// Background
		ctx.fillStyle = "rgba(100, 149, 237, 0)";
		ctx.fillRect(0, 0, 200, 30);

		// Text
		ctx.fillStyle = "rgb(0, 100, 0)";
		ctx.font = "20px Impact";
		ctx.fillText("Users online: " + count, 10, 20);

		return canvas.toBuffer();
	}
});

function handleRequest(req, res) {
	switch (req.url) {
		case '/':
		case '/index.html':
			showDemoPage(req, res);
			break;
		case '/online.png':
			showImage(req, res);
			break;
		default:
			show404(req, res);
			break;
	}
}

function showDemoPage(req, res) {
	res.writeHead(200, {'Content-Type': 'text/html'});
	res.write("<h1>Users viewing this page:</h1>");
	res.write("<img src=\"/online.png\" />");
	res.write("<h5>(probably won't work on IE or Opera)</h5>");
	res.end();
}

function showImage(req, res) {
	// If this image is not embedded in a <img> tag, don't show it.
	if (!req.headers.referer) {
		res.writeHead(403, {'Content-Type': 'text/html'});
		res.end("You can't view this image directly.");
		return;
	}

	// Create a new client to handle this connection
	clients.add({
		req: req,
		res: res
	});
}

function show404(req, res) {
	res.writeHead(404, {'Content-Type': 'text/html'});
	res.end("<h1>not found</h1><br /><a href=\"/\">go home</a>");
}

// Ready, Set, Go!

var clients = new Clients();
http.createServer(handleRequest).listen(config.port, config.host);

console.log("Started.");

(00:12 EST) UPDATE 1: I’m pretty sure I fixed a bug where 6% of connections never seemed to “disconnect”. These changes have been reflected in the code shown above, as well as in the file linked just above that. Here’s what I changed:

--- a/standalone.js
+++ b/standalone.js
@@ -28,6 +28,7 @@ var Client = Backbone.Model.extend({
 		console.log("Page opened:", req.headers.referer);

 		res.on('close', _.bind(this.handleClose, this));
+		req.on('close', _.bind(this.handleClose, this));
 		this.sendInitialHeaders();
 		this.set('updateinterval', setInterval(_.bind(this.sendUpdate, this), config.updateInterval));
 	},
@@ -79,6 +80,9 @@ var Client = Backbone.Model.extend({

 	// Handle a disconnect
 	handleClose: function() {
+		if (this.get('closed')) return;
+		this.set('closed', true);
+
 		console.log("Page closed:", this.get('req').headers.referer);
 		this.collection.remove(this);
 		clearInterval(this.get('updateinterval'));

(13:54 EST) UPDATE 3: Well, turns out that didn’t fix the bug. It’s currently showing around 130 users online, but according to real-time Google Analytics, it’s only 30 users at the moment. I’m having trouble figuring out how to fix this, because I’m already binding to the only two events Node.js offers for detecting an HTTP disconnection. If you see where the bug might be, I’d certainly appreciate you leaving a comment.


I can see some neat uses for this type of image, since content can be made interactive without the use of client-side scripts. Interactivity can be introduced by having links with the server returning HTTP 204 No Content, such as this link (provided by httpstat.us). The client won’t go to a new page, but the server will be aware of the client’s request and can perform an action. With this, real-time tic-tac-toe or even hang-man is possible.

Just some thoughts.

You can follow a discussion of this post on Hacker News.

(03:28 EST) UPDATE 2: Through experimentation, I just discovered that unfortunately, it is not possible to combine HTTP 204 links with image streams, as I suggested in my last paragraph. Upon clicking any link, the browser stops loading all resources, including streams. This is before it even knows that the result status will be 204. What a bummer.

    14 thoughts on “Real-time user count without any Javascript

        1. Drew Gottlieb Post author

          Thanks!

          I’m actually a bit worried if there’s a leak-ish issue here.
          On line 30 you can see where I bind to the response object’s ‘close’ event. Perhaps I should have bound it to the the request object’s ‘close’ event as well.

          At the moment, I don’t really have any way of knowing if the number we see right now is true or not. I’ll just have to wait until tomorrow and hope that it goes all the way back to zero :P

      1. Drew Gottlieb Post author

        Well I’m binding a function to the request and response objects’ “close” event, so that I know when the connection was terminated (i.e. the user navigated away from the page).

        But instead of doing: res.on(‘close’, this.handleClose), I use underscore’s _.bind() function. Why? Well if I did it the way I just stated, when the handleClose() function is finally executed, its “this” object won’t necessarily refer to the Backbone model that it’s a part of. _.bind(func, obj) returns a new function with the first parameter (the function)’s context being the second parameter. I’m probably not explaining it all that clearly, so have a look at this: http://underscorejs.org/#bind

        And here’s the doc for the “close” event: http://nodejs.org/api/http.html#http_event_close_2

        I hope I answered your question well enough. It’s about 4:00am here and I probably shouldn’t even be typing at this point. ;)

    1. Alex Nichol

      In a situation without Javascript, this is a brilliant idea. I also had no idea that any browsers supported JPEG streams directly through HTTP; I’ll definitely have to check this out and play with it later. As far is internet explorer goes, I wouldn’t worry about it. If someone opens your website in IE, chances are it’ll look so bad that they’ll leave before the MJPEG stream would have loaded anyways ;) .

      The only other thing I could maybe think of to achieve the same goal would be a sort of circular redirect loop where a user would request an image, be redirected to another image, which then redirects to another, etc. However, I’m sure most browsers are smart enough to prevent against this.

      1. Drew Gottlieb Post author

        Yeah, not only does Chrome, for example, give up after several HTTP redirections, but it wouldn’t work for an image either since the message body would be ignored with HTTP 302 responses.

    2. Tatyana Dettinger

      Yes, there are some amazingly creative talents in blog land! We all feed off of each other’s inspiring work. It’s a chain reaction!

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">