SLAP – Simple Lightweight Announcement Protocol

Today, to only two close developer friends, I’m releasing a first prototype of a new project called SLAP (for Simple Lightweight Announcement Protocol). We’re going to begin multi-server testing of it as soon as possible. If you’re not a developer, you may not immediately grasp exactly what SLAP is all about, but I’m hoping it will help all of us establish our own Social Networks of One, as Jeff Pulver calls it, and not be slaves to the whims of services like Twitter or Facebook, that now determine when and how we can communicate.

SLAP is a project I’ve been developing on and off for a couple of years. SLAP provides a very minimal and fast way to alert content or status subscribers that new or updated information is available, without them having to poll constantly for it.

By eliminating polling we benefit both the subscriber and the publisher. The subscriber gets informed of new content immediately, with no polling period latency and no resources of computing or bandwidth are expended between announcements. Likewise, the publisher does not need to provide sufficient bandwidth and computing resources to respond to polling requests, the vast majority of which result in no new information.

Unlike HTTP and XMPP, SLAP is connectionless, employing UDP (User Datagram Protocol) as its underlying transport mechanism. This is why no significant server or network resources are used between announcements. Although XMPP provides the same immediacy of delivery and elimination of polling, it requires long-lived connections to be maintained even if the frequency of announcements is very low, e.g., once a week. HTTP does not require long-lived connections, but instead requires establishing and terminating much TCP connection overhead in a short amount of time to deliver a very small amount of data.

The SLAP model

SLAP defines announcements in terms of feeds and posts to those feeds. Each feed is identified with a URI and each post with a perma-link URI. Further, each announcement is uniquely identified independently of the feed and post it is related to. This allows many announcements of the same feed and post. For example a post may be initially created, then edited many times, or commented on many times. Each edit or comment can result in a new announcement of a change to the post.

Publishers and Subscribers

There are two basic roles in the use-case model for SLAP; the Publisher and the Subscriber. The Publisher creates new feed content and emits a SLAP event every time that feed content changes that goes to every Subscriber. The Subscribers receive the announcements and respond with an acknowledgement. The announcements and acknowledgements are all delivered using UDP, which means no connection overhead for either the Publisher or the Subscriber. The protocol is tolerant of off-line Subscribers and noisy, unreliable networks. Publishers retry unacknowledged announcements at a fixed period and for a limited amount of time. One of the strengths of SLAP is that a missed announcement is not the end of the world. The Subscriber can periodically check the Publisher for all feed content and discover a missed post that way.

Strictly speaking, SLAP is not intended to deliver content, but only to announce that new or modified content is available on a feed and to provide the URIs for the feed and the post that allow the subscriber to fetch the content. SLAP does provide for a content summary that may in many cases be all the content there is. Microblogging or automated system status updates would tend to use this feature. In these cases, an attribute of the message could show that the summary is the entire post content. This is an optimization that enables a subscriber to avoid making a content fetch when the summary is all that there is.

Having the publisher be responsible and obligated for content storage allows a subscriber that has been offline for an amount of time to fetch any posts it missed simply by reading the feed as it normally would.

Separation of subscribing and fetching from announcing

The core SLAP messaging services deliver and acknowledge announcements, but play almost no part in establishing or terminating subscriptions. Nor do they specify how fetching of post content happens. Subscriptions can be established in any number of ways, as long as the required information for announcement delivery and authentication is provided. The subscription protocol is an open development issue, and your comments and suggestions are welcome. Personally, I see a simple extension to RSS as being the most obvious route. This would allow common blogging or microblogging applications like WordPress or Twitter to add announcement support with a relatively small modification to their current RSS support. RSS feeds could allow SLAP-enabled subscribers to discover the SLAP parameters and automatically connect to those feeds providing SLAP announcements.

Authentication

Any protocol that “pushes” information to a destination opens a possible hole for spammers. One of the challenges to making SLAP work is enabling subscribers to authenticate incoming announcement messages as to their source and to do it relatively quickly. This first version of SLAP uses a simple MD-5 hash of the message contents with a subscriber generated key that is exchanged at subscription time, ideally using secure communication.

Try it, you’ll like it.

SLAP can be used just for publishing, just for subscribing, or both. You’ll need to run it on a machine that you can open the firewall on, and receive UDP messages on ports 14947 for the subscriber and 14948 for the publisher. Or you can use any SLAP server that you’re given access to.

For what it’s worth, I think journalists are unnerved and defensive because one of the mediums that they’re published on now is open to anyone to publish.  They seem to feel that this is unfair. Because their access to the web is controlled, they seem to feel everyone’s should be.

Mr. Schorr, on that same NPR show, said that the internet was like a market or bazaar where anyone can come and shout anything they want and be heard by anyone who cares to listen. This is a particularly apt analogy. If we apply it to journalists on the internet, we see that it’s like making them stand on a milk crate and announce the news on the street corner just like everyone else, whereas before they had special arrangements. They spoke from a balcony with a loudspeaker.

And that’s the inversion that I think not everyone appreciates. It’s not so much that we, the bloggers have invaded the journalists’ turf, it’s rather that they’re now forced to operate on ours. A medium used to come with a certain implied or built-in credibility and certainly privilege. If you got to speak, it was sort of assumed that at least you had your facts more or less straight.

But access to a medium does not necessarily equate to credibility and really, it never has. Anyone with sufficient money can print their own newspaper. But just because something is printed on sheets of paper that look like a newspaper doesn’t mean it’s true. Trustworthy journalists and news institutions earned their stripes over the years by doing their jobs well and being proven out by facts and people’s real-world experiences.

I think journalists should believe in their own credibility a little more and realize that most of us out here reading things on the internet value reputation and track record just like we did before. Sure, there are still plenty of people who’ll choose the equivalent of the supermarket tabloid or Fox News, but you never had them anyway. It not that you’re standing on a street corner with everyone else that matters, it’s what you say when you get there.

establish a full environment for an XMPP based microblogging service. Environment means the full stack necessary beginning from protocol and API definitions to server side and client side implementations.

I am hoping that this project will also be able to make a distinction between publishing of content and notifying subscribers of the publishing event. This is a generally useful function that transcends microblogging. It would be a great boon to live-bloggers, news services, live-streaming video or any publishing endeavor where time is a factor.

I’ve written in this space previously about the idea of a personal publishing platform. OpenBirdcage may give us the opportunity to decouple the notions of creating and posting content from the notion of notifying others that the new content is available.

In this decoupled model, you could choose a notification provider separate from your publishing provider, although certainly the two might be offered by the same site. In addition, you could choose multiple notification providers so if one failed, subscribers could simply fall over to one or more backup site.

This model would allow not only microblogs, but any web publisher to very rapidly notify subscribers of new or changed content without having to poll. Your personal identity and/or publishing site would present XRDS records allowing subscribers to discover your notification provider accounts, which could be totally distinct from your publishing provider.

XMPP publish-subscribe plus Atom is well-suited to this model. It would allow small content like microblog posts to be sent right in the notification, while links would be sent for larger content.