SLAP – Simple Lightweight Announcement Protocol

Today, to only two close developer friends, I’m releasing a first prototype of a new project called SLAP (for Simple Lightweight Announcement Protocol). We’re going to begin multi-server testing of it as soon as possible. If you’re not a developer, you may not immediately grasp exactly what SLAP is all about, but I’m hoping it will help all of us establish our own Social Networks of One, as Jeff Pulver calls it, and not be slaves to the whims of services like Twitter or Facebook, that now determine when and how we can communicate.

SLAP is a project I’ve been developing on and off for a couple of years. SLAP provides a very minimal and fast way to alert content or status subscribers that new or updated information is available, without them having to poll constantly for it.

By eliminating polling we benefit both the subscriber and the publisher. The subscriber gets informed of new content immediately, with no polling period latency and no resources of computing or bandwidth are expended between announcements. Likewise, the publisher does not need to provide sufficient bandwidth and computing resources to respond to polling requests, the vast majority of which result in no new information.

Unlike HTTP and XMPP, SLAP is connectionless, employing UDP (User Datagram Protocol) as its underlying transport mechanism. This is why no significant server or network resources are used between announcements. Although XMPP provides the same immediacy of delivery and elimination of polling, it requires long-lived connections to be maintained even if the frequency of announcements is very low, e.g., once a week. HTTP does not require long-lived connections, but instead requires establishing and terminating much TCP connection overhead in a short amount of time to deliver a very small amount of data.

The SLAP model

SLAP defines announcements in terms of feeds and posts to those feeds. Each feed is identified with a URI and each post with a perma-link URI. Further, each announcement is uniquely identified independently of the feed and post it is related to. This allows many announcements of the same feed and post. For example a post may be initially created, then edited many times, or commented on many times. Each edit or comment can result in a new announcement of a change to the post.

Publishers and Subscribers

There are two basic roles in the use-case model for SLAP; the Publisher and the Subscriber. The Publisher creates new feed content and emits a SLAP event every time that feed content changes that goes to every Subscriber. The Subscribers receive the announcements and respond with an acknowledgement. The announcements and acknowledgements are all delivered using UDP, which means no connection overhead for either the Publisher or the Subscriber. The protocol is tolerant of off-line Subscribers and noisy, unreliable networks. Publishers retry unacknowledged announcements at a fixed period and for a limited amount of time. One of the strengths of SLAP is that a missed announcement is not the end of the world. The Subscriber can periodically check the Publisher for all feed content and discover a missed post that way.

Strictly speaking, SLAP is not intended to deliver content, but only to announce that new or modified content is available on a feed and to provide the URIs for the feed and the post that allow the subscriber to fetch the content. SLAP does provide for a content summary that may in many cases be all the content there is. Microblogging or automated system status updates would tend to use this feature. In these cases, an attribute of the message could show that the summary is the entire post content. This is an optimization that enables a subscriber to avoid making a content fetch when the summary is all that there is.

Having the publisher be responsible and obligated for content storage allows a subscriber that has been offline for an amount of time to fetch any posts it missed simply by reading the feed as it normally would.

Separation of subscribing and fetching from announcing

The core SLAP messaging services deliver and acknowledge announcements, but play almost no part in establishing or terminating subscriptions. Nor do they specify how fetching of post content happens. Subscriptions can be established in any number of ways, as long as the required information for announcement delivery and authentication is provided. The subscription protocol is an open development issue, and your comments and suggestions are welcome. Personally, I see a simple extension to RSS as being the most obvious route. This would allow common blogging or microblogging applications like WordPress or Twitter to add announcement support with a relatively small modification to their current RSS support. RSS feeds could allow SLAP-enabled subscribers to discover the SLAP parameters and automatically connect to those feeds providing SLAP announcements.

Authentication

Any protocol that “pushes” information to a destination opens a possible hole for spammers. One of the challenges to making SLAP work is enabling subscribers to authenticate incoming announcement messages as to their source and to do it relatively quickly. This first version of SLAP uses a simple MD-5 hash of the message contents with a subscriber generated key that is exchanged at subscription time, ideally using secure communication.

Try it, you’ll like it.

SLAP can be used just for publishing, just for subscribing, or both. You’ll need to run it on a machine that you can open the firewall on, and receive UDP messages on ports 14947 for the subscriber and 14948 for the publisher. Or you can use any SLAP server that you’re given access to.

It depends on how the UI for subscribing works. If, as with a feed reader, the subscriber enters the URL of the person they wish to follow, authorization is unnecessary. The user has made their intention known to their own account, and can simply request the posting server to send updates.

However, if the the subscriber subscribes by visting the site of the poster, and the poster allows unrestricted access, then there must be a mechanism for the subscriber to tell his own site that he wishes to subscribe to the poster’s updates. This is where OAuth comes in, and it is indeed a well-matched use-case for OAuth’s functionality.

Apologies to Evan at Identi.ca.

Square peg, round hole

The OpenMicroBlogging Specification (OMB) contains a section on the use of OAuth to authorize a remote service to post to a user’s local service. After spending some time studying the OAuth and OMB specs, it seems to me that this is not the use-case that OAuth was designed to solve. A square peg in a round hole, you might say.

This is not to say that authorization is not needed for allowing post-sharing, only that OAuth was designed to address a different situation. The simplest way I can characterize the difference is that OAuth applies to a “one human, two services” problem and microblogging post-sharing is a “two humans, two services” problem.

OAuth’s intended use-case

So, what problem does OAuth fit? OAuth is intended to solve the problem of a person who has content or resources on Service A and wants to allow his/her account on Service B to access those A resources – without giving Service B his/her login credentials for Service A. Note well that it is the same person having accounts on both A and B. This is the “one-human, two services” aspect.

So, for example, let’s say you have an account on Twitter and want to use one of the many alternate Twitter clients out there, e.g., Twalala. You have an account with Twalala and would like to be able to access your account on Twitter without having to fork over your Twitter password. Using OAuth, you would be able to, in essense, tell Twitter to allow your account on Twalala.com to perform various read or post functions with your Twitter data. By the way, Twitter has been promising for months to implement OAuth for exactly this purpose. Again, note that there is only one person with accounts on both systems.

OMB post-sharing use-case

Now, let’s look at the case of two people on two different OMB-compliant sites who want to share their posts, either unilaterally or bilaterally. Let’s say I have an account “joecascio” on site subscriber.com, and my friend has an account “johnsmith” on poster.com and I would like to follow John’s updates. Note that it should not be necessary for either me to have an account on poster.com nor John to have an account on subscriber.com in order for me to “see” his updates. Indeed, this is the whole point of federated microblogging; that we can share posts without having to have accounts on many different services. So simply from this fact alone, OMB post-sharing is clearly not the “one person, two accounts” use-case. If post-sharing worked by requiring John to have an account on subscriber.com (the second of the “two services”) and then authorizing subscriber.com to let his Poster.com account post there, then yes, that would be the OAuth use-case. But it isn’t!!

So, what is an appropriate protocol or process for enabling and controlling post-sharing? I don’t have the complete answer, and I’m hoping that this post will cause some of the many smart people out there to suggest some. But I can state at least some of the problem characteristics, requirements or constraints.

Post-sharing is subscribing, not cross-posting

When I register my interest in someone else’s microblog posts, when I “follow” them, it means I want to subscribe to their post stream, just like I would subscribe to their traditional blog. If it were a traditional blog, I would do this by polling their RSS feed which is a simple GET by the subscriber. It does not involve the poster initiating a transaction to “send” me anything. But since microblogging implies a much lower latency, which would be inefficient to perform by polling, OMB introduces an optimization. The service on which the post was originally made proactively sends the new post to all subscribers. Great! We can now be notified that a new update is available within a few second without having to poll every few seconds.

BUT this is point of my observation. I would argue that this proactive send is merely an artifice of the optimization and does not change the fundamental relationship of poster and subscriber. From that basic user point of view, the fact that the poster initiates the data transfer is immaterial. The subscriber is still simply seeing something they expressed interest in reading.

This is where I believe OMB goes astray. It lets an optimization detail obscure the higher level concept. Following someone’s posts means I want to see those posts in my view, but it does not require that either party perceive it as cross-posting to my site. I could equally accomplish the same viewing of that post by fetching it on demand, albeit not as efficiently.

Subscribing is implicit authorization

Keeping firmly in mind the fundamental concept of poster and subscriber, what if any authorization needs to happen when I request to follow ‘johnsmith’? I would say that the very fact I am requesting to see John’s posts is an implicit authorization that he can send them to me. Whether he wants to let me see them is a different story and the subject of another post. Let’s assume he will let me see his posts and we want to use the OMB optimization of his service (poster.com) proactively sending them to me.

When I request to subscribe, I need to send a URL of where to send the notifications and a means to verify that the sender is who I granted the privilege to.  This could be accomplished to different degrees of security. A fairly loose method would be for the subscriber to send a callback URL in the request that encodes a UID for the poster’s account. If the request transaction is secure, we can be reasonably certain that only the the “real” poster will be sending POSTs to that URL. Of course, this does not prevent the poster from sharing the UID, but the subscriber can always simply invalidate the URL and stop receiving the notifications.

In fact, this is pretty much how the RSS cloud API works and this could be a more proper model for OMB post-sharing.

What do you think?

I would be very interested to hear what others have to say about this post. One thing it does not cover is the complementary function of a subscriber GETting, rather than being sent an update. It also does not cover the perfectly matched use of OAuth to allow a user to grant access to other of his/her services just as described in the Twitter/Twalala example under “OAuth’s intended use-case”.

establish a full environment for an XMPP based microblogging service. Environment means the full stack necessary beginning from protocol and API definitions to server side and client side implementations.

I am hoping that this project will also be able to make a distinction between publishing of content and notifying subscribers of the publishing event. This is a generally useful function that transcends microblogging. It would be a great boon to live-bloggers, news services, live-streaming video or any publishing endeavor where time is a factor.

I’ve written in this space previously about the idea of a personal publishing platform. OpenBirdcage may give us the opportunity to decouple the notions of creating and posting content from the notion of notifying others that the new content is available.

In this decoupled model, you could choose a notification provider separate from your publishing provider, although certainly the two might be offered by the same site. In addition, you could choose multiple notification providers so if one failed, subscribers could simply fall over to one or more backup site.

This model would allow not only microblogs, but any web publisher to very rapidly notify subscribers of new or changed content without having to poll. Your personal identity and/or publishing site would present XRDS records allowing subscribers to discover your notification provider accounts, which could be totally distinct from your publishing provider.

XMPP publish-subscribe plus Atom is well-suited to this model. It would allow small content like microblog posts to be sent right in the notification, while links would be sent for larger content.

One thing that’s become obvious to me after working on and thinking about Distributed Microblogging is that all forms of messaging are essentially the same. The only differences between email, IM, chat and microblogging are quantitative not qualitative. There are only matters of degree in length, asynchrony and number of recipients. While it is true that sometimes a matter of degree results in a defacto matter of kind, I’m convinced there is some kind of Grand Confluence of these different messaging modalities in our future.

I didn’t include blogging as messaging because the essence of a message is that the sender initiates the transfer. So even though you might subscribe to a blog using RSS, feed readers still operate by polling, i.e., periodically checking for something new. So blogging is still basically a “pull” operation, where messaging is a “push”.

But blogging and messaging can be thought of as similarly implemented if we separate two notions: message content and message notification. Think of sending a message as two operations. First, I create and store the message content on a place I’ll call my Personal Publishing Site or PPS. Second, I send out a short notification message to the recipients, with a link or key back to the stored post on my PPS. Third, using the link or key I provided in the notification, the recipients call back and get the content from my PPS. Now, in the case of small messages like IM, chat or microblogging, it would be a time and bandwidth saver to just send the message body along with the notification. One correlation that seems pretty consistent is that short messages like IM, chat or microblogging tend to get sent more frequently and with a shorter delivery requirement than say, email or blog posts, which can languish for days without being read.

What if all messaging operated this way? What if, instead of going to different sites to talk to friends there with different tools and capabilities, we created and sent (published and notified) all messages using our own Personal Publishing Site? What if community or other types of sites could receive our PPS messages and post copies of them or links to them?

We have a glimpse into the possibilities of this world with Identi.ca and the open source software that powers it, Laconica. What if everyone ran their own laconica service? That would be like a Personal Microblogging site. What if our OpenID page contained meta-data about what laconica server to use to contact us? What would that mean for the notion of community sites? Would they become more like simple mailing lists if you didn’t have to “go” there to communicate with your friends who might also be friends there? In this model of the world, we’d have discussions by a sort of Dueling Banjos publishing. I publish a message which you read, then you respond by publishing a message that I read. Depending on how subscription works only you and I might see the messages, or they might be seen by a wider group of our subscribers or anyone, sort of like an @ conversation on Twitter.

ps: The DISO project is attempting to address some of these issues, if you want to pop over there and do a little reading.

So I’ve been putting together the following list of what I consider to be important features that a good microblogging function should have. In most cases, I’ve approached it from how it should appear to the user. If you are coming to DevHouseBoston on Sunday, you might want to read this list over and come prepared to argue the merits of different points, add your own, or if you’re a UI designer, contribute to a discussion of how to design a UI that incorporates these features.

Remember that one of the benefits of an open system is that there can be many different UIs to choose from!

So, following, is the list in a very raw and unpolished form.

Note: DMB means Distributed Microblogging.

Feature list for dmb UI

Omnipresent replies, direct messages tabs or buttons.
Tags on other microblogs, on posts, on other tags?

Direct support for conversations, replies, etc.
Main page. Like twitter. Update input box at top, but maybe with separate scrolling list of stream.
Multiple streams showing, replies, all, selected person’s archive, including user’s archive.
“Hide” check boxes for each user. Clicking it removes their updates from stream list, and places their name in the a “hidden” list for easy reinstatement.

Also would be able to show/hide certain groups.

Conversation support. Each tweet would have a “reply” button. Clicking it would add the @user_name to the beginning of the update, and also make a database entry linking the update to one or more previous updates. The @’d user names would be like the “to” list on an email. Each update could have multiple other updates it was in-reply-to. This is a many-to-many relationship. An update can be in response to multiple other updates, and clearly an update can have many responses to it.

Updates have GUIDs. Required by IMPP.

Tabs for different subsets of users = from Mike Gaines. “tag-tab”

Separate input area for different tag-tabs.

Topics for User experience
Conversations. What can you do? How are they shown?
Multiple addressees (like email?)
Latest on top or on the bottom? User pref?
Classes of messages
Direct
Narrowcast (to a group tag?)
Broadcast (or is this simply to all tags?)
Temporary silencing (show/hide checkbox) So I can turn off someone who is usually ok, but is just ranting or something.

Ad-hoc grouping. Drag users into a tab where you see only their tweets.
Public ad-hoc groups. All your followers can see tweets. You just filter locally.
Private ad-hoc groups. Just the people in the group see the tweets.

General question.. will adding private groups destroy some of the vibrancy and serendipity of Twitter?

How to present tagging in the UI so it’s intuitive with what it does for the user, as opposed to a geeky abstract function?

What can be tagged, and hence grouped? Users, convos, even other tags? Locations? languages? specific tweets having to do with a certain subject? others?

Universally unique tags vs. private tags. UUtags would be very helpful for organizing large tagging efforts. But, how to agree on a tag for say, “SocialMediaBreakfast8″?

theRQ would be great to be able to send/post an out “out of stream” message to someone that they get upon login- wouldn’t age off. (subject to spam abuse?)

Search – Let Google do it? What about private mblogs? Can individual servers be expected to support a general search capability? Thru XMPP? Thru HTTP?

Show me when I have new, unread DMs.

Multiple addressees for DMs. Now it’s beginning to look like email.

Show me when I have new, unread messages.
Show me when I have new, unread messages from specific people or groups. (on that tag-tab?)

Auto-update (ala chat) vs. manual update.

Rich content. Photos, audio. (How to transmit to mobile devices?)

How to add these features without sacrificing the simplicity?

A few lines when you request to follow, so you can explain why. Either that or a listing of common friends on different sites.

What about the Twitter community? Can it be made part of DMB? Can we build an interface to Twitter? Will Twitter participate in DMB?

What about other communities that start up? Twitter has very amorphous sub-communities.

Implementation questions:

Use XMPP archiving or external? (some xmpp archiving options not applicable, like off the record, although they might be of interest)

How to do multiple servers?

How to map mb posts onto “collections” in xmpp archiving. Another reason to use external archving?

If we did external archiving it would relieve the burden of archiving on servers that could be used. Although they’re more likely to support archiving than pubsub.

I had the pleasure of speaking at BarCampBoston3 this past Saturday in Cambridge, MA on the topic “Distributed Twitter”. This was a well-attended event and the geek energy was high! BarCamps attract and cater to hard-core coders and implementers. Not many marketing, PR or media types were in evidence, but there were quite a few startup CEOs there looking for help.

One particularly useful session for me was at the start of the day, when Matt Douglas, CEO of MyPunchBowl.com, brought together companies looking for help with developers looking for work. It was very informal, we all got 10 seconds to say who we were and what we were looking for. I made a couple of promising connections there, so that in itself was worth the 90 minute drive from my home in Connecticut to Cambridge. Thanks, Matt and MyPunchBowl! This kind of Hook-me-up session should be a part of every BarCamp.

I was pleased that my session on Distributed Twitter attracted a packed room. Unfortunately, I completely zoned and missed the fact that sessions were only 1/2 hour, so I had to cut things short, but I did meet some interesting people who were interested in trying to get a project off the ground to actually build something.

I had some interesting follow-up discussions later in the day with people who were at my talk, but unfortunately, did not get to tell all at the meeting about the Distributed Twitter Google Group. Probably no biggie since I’m going to have to change the name of the group anyway, to avoid the inevitable cease-and-desist letter from Twitter.com’s lawyers for using their name.

One of the big reasons I went to BarCamp was to try to connect with mobile developers since a big part of decentralizing a microblogging platform is to provide the mobile connection that Twitter has. Many people use Twitter almost exclusively on their phones. I did meet two young fellows who seemed very interested in creating a mobile app to interface directly to a microblogging platform. Their advice was to skip SMS for demo purposes and simply use the mobile IP connectivity. Their POV was that SMS will go away in due time to be replaced by the more flexible and extensible IP protocol directly to apps on the phone. I’m not sure I buy the argument that you don’t need to have an SMS connection. SMS will be around for a while to come yet, and not having it could hamper uptake severely.

I had other interesting conversations regarding the use of XMPP as the “carrier” protocol for microblogging services. It was suggested to use Google’s gTalk service to experiment with since it’s a fully compliant (I think) XMPP end-point.

I continue to have some challenges with XMPP proponents not fully appreciating that microblogging is more like blogging than instant-messaging or chat. I’ll confess I don’t understand the differences completely myself, but I do see that there are some and that they are significant to a proper implementation. For instance, microblogging does not have the requirement or notion of presence. As a user, you don’t need to be “present” to get all the updates that people may send. You should be able to go back at any time and get the complete history of a microblog’s updates, just like it was a blog. Related to presence is the notion that senders of content aren’t responsible for keeping a permanent copy around for receivers to reference and search. More about this at a later time, but one shouldn’t have to rely on being “present” or having your receiving server make a copy of everything that a microblog sends to have a complete record. The sender should be responsible for keeping a permanent, permalinkable copy.

So, all in all, BarCampBoston3 was definitely worthwhile but I could have done better had I taken two consecutive sessions, which is allowed, and not have been so pressed for time.

Coming up…

In the weeks to come, I’m scheduled to do two more presentations about Distributed Microblogging.

1. IgniteBoston3 - May 29th, 6 pm at Tommy Doyle’s in Harvard Square, Cambridge, MA. This will be a quick, 5 minute “drive-by” preso just to introduce people to the topic. I’ll probably just get enough time to describe what it is, why I’m involved and to pimp preso #2 (next) and refer people to the Google group. If you’re interested in coming, be sure to register because they always fill up!
2. Berkman Center Thursday Night Meeting - June 5th, (time: tba), at the Berkman Center for Internet and Society, Harvard University, 23 Everett St. Cambridge, MA. I’ll be speaking on Distributed Microblogging. I’m not sure, but there may be other presenters as well. This is my first time at a Berkman Thursday Night meeting and I’m very excited.

Ok, so let’s talk about the hard bits of doing Distributed Microblogging. It’s easy to envision a multitude of servers exchanging microblog posts, and a UI that simply arranges the posts in chronological order. By the way, chronological order is easy to do if all the servers are synched with a time standard like nist.gov, and most are. But the hard bit is making it work in a way that performs well and scales to large populations of both microblog posters and readers. I’ve been thinking of some different alternatives for this, which I’ll lay out here. As always, your thoughts are welcome.

Performance

So, what do I mean by “performs well”? Well, microblog (ie, Twitter) updates happen much more frequently than what we’d consider traditional blog posts, but not quite as fast as instant messenger or chat updates. And microblogs don’t have the notion of presence attached to them. You don’t worry about whether a Twitter poster is “on line” at any particular time, although you may deduce that from the rapidity of their responses to you.

To put a number on it, I’d say that a microblog post notification should be transmitted in less than a minute, ideally in a few seconds, although a delay of up to 5 minutes is not that objectionable. I frequently will ignore my Twitter feed for many minutes, sometimes hours. I have no expectation that I will see someone’s posts immediately (i.e., less than a second) nor do I care to. Microblogging is a river of updates. You don’t expect to see every single one.

Size

Now let’s talk numbers of followers and following. A typical Twitter user has 100 or fewer people that they follow, and a similar number that are following them. But the edge cases are far bigger. Someone like Robert Scoble (@scobleizer) or Chris Brogan (@ChrisBrogan) follow literally thousands of other users, and have even more following them.

Ok, so we need to be able to update thousands of users in nominally less than a minute, but at the exreme less than five minutes. If you don’t buy that, please leave a comment explaining why these limits are not realistic.

Alternatives

RSS feed polling

The simplest solution would be polling, which is reader initiated. Followers would poll the feeds of the microblogs they were interested in. It seems to me that polling has to be discarded as a solution except for occasional or retrospective uses. I think a solution should include RSS feeds but it seems obvious to me that for someone with 100 friends to poll those RSS feeds every minute, or ideally faster because they might post something is a stupendous waste of bandwidth.

On the sending side, if that same person had 100 followers, each of those followers would be polling the person’s RSS feed every few seconds causing a very high server load. Now multiply that by however many people’s accounts are hosted on that server and the problem quickly blows up.

Notification

A much more efficient solution would be for senders (microblog posters) to notify followers when a new post has been made, and perhaps to proactively send the new content in the notification message. Then, resources are used only when there is actual traffic to send. Both senders and receivers are quiescent when no one is posting. So what are the possibilities for implementing notification? I see the following.

1. RSS cloud API – a little-known part of the RSS specification, the cloud element allows a feed to publish a web-service address that readers of the feed can register with to be notified of changes using a SOAP or xml-rpc call.
2. Jabber (XMPP) channels between DMB servers to carry notifications and content.
3. UDP notification with http callback. UDP is lightweight for both senders and receivers. No open connections are required between senders and receivers. It’s sort of like RSS cloud, but narrowly and specifically designed for DMB, as opposed to generalized RSS.

RSS cloud API

The cloud API was specifically designed with this purpose of notifying readers of content updates. Its original intent, judging from the RSS 2.0 spec was to allow actual client feed readers to register with the cloud. In the case of DMB, it would be cooperating servers that would register for notification with each other.

The problem with RSS cloud is overhead. Microblog entries are tiny and frequent compared with blog entries or traditional site updates. To require a follower server to read an entire RSS document to get 140 characters of content, and have this happen every few minutes when the poster updates would be inefficient to say the least. In addition, there is experience with the cloud api that indicates just the HTTP session overhead for notifying many users becomes intolerable, although this was from the perspective of actual clients being notified as opposed to clients’ servers being notified.

Jabber (XMPP)

Jabber is a very tempting candidate for this application, and has been getting quite a bit of discussion in the development community of late. Here’s an example. The advantage to Jabber is that it maintains open sessions between servers, which eliminates the session setup/teardown overhead, and allows for almost instantaneous notification of all “following” parties.

But this may also be a disadvantage in situations where there are hundreds or thousands of “followers” for a single sender. IM or chat is typically one-to-one, or one to a few, but microblogging is frequently one to hundreds or one to thousands. I am not familiar enough with Jabber servers in actual practice to know what their performance or connection limitations are. Anyone with Jabber implementation or operational experience is strongly encouraged to comment.

Jabber’s concept of presence could be used to keep the number of messages by only requesting messages updates when a user is actually logged into his/her microblog system. What’s interesting about this notion is what “logged in” means. Microblogging, at least the way Twitter works, does not really require the concept of presence. For instance, you can be “logged in” to Twitter, but dormant for hours not getting any updates until you request a page refresh.

In fact, one key aspect of microblogs that differentiates them from IM or chat is that they don’t typically “auto-update”. And rather than this being a disadvantage, I-and I think many others-find this on-demand update to be much more useful than a streaming IM or chat window. It’s really more like reading small blog posts. I read when I want to, not when someone else decides to say something. So, using the presence capability without auto-updating will require a little clever UI design to, in a sense, auto-logoff the user when their web page hasn’t been refreshed in a certain period of time. This then, will also require the ability of followers to query the senders they follow for “back posts”, so they can see what happened in the past without keeping a client logged in all the time to save all the posts.

UDP for notification

In the past year, I did some work on a revamped email protocol I called IMTP that uses small UDP messages to notify receiving servers that the sender has some traffic for them. The receiver then calls back to the sender using TCP to get the message body. This was based on Prof. Daniel Bernstein’s Internet Mail 2000 proposal several years ago. He proposed, quite rightly in my opinion, that mail senders should bear the burden of storing the message contents, not the receivers, and that mail content should only be sent when a receiver actually wants to read it.

The advantage to UDP is that it is very light weight for both the sender and the receiver, not requiring any session overhead or setup/teardown. If used for microblogging, UDP notification messages could take the place of the continuously open TCP sessions that Jabber employs, thus reducing the session resource allocation load on both ends.

Now, of course, the issue with UDP is that it is not guaranteed to be delivered. The IMTP service we built would retry the UDP message at some frequency until the receiver called back to either fetch or reject the message.

Conceptually, using UDP messages with a 140 character payload and a message number would fit a microblogging application very well. Because microblogging has no expectation or requirement of presence or real-time delivery like chat and IM do, a dropped UDP message is not a tragedy. If the sender retried even once or twice per minute, that would be plenty of timeliness for microblogging. Plus, if the UDP messages carry a monotonically increasing message number, the receiver can know if they’re missed a message and simply call back to the sender to get it. The microblogging UI can reconstruct the sequence easily when the missing pieces, if any, finally come through.

A notification system using UDP would seem to minimize the resource requirements on both senders and receivers, and can perform the same kind of message fan-in/fan-out optimization that Jabber could. In other words, a given microblog update body needs to be sent only once to any server, regardless of how many followers there may be on that destination server.

Comments, please!

I am very interested in what others think and have to say about these issues. This problem of efficient and timely-enough notification, it seems to me, is the tough nut to crack for a good solution to microblogging.

By the way, I am going to do a session on Distributed Twitter (or Microblogging) at BarCampBoston3 on May 17, 18. We are going to try to bring in some mobile technology folks to discuss the other really interesting issue with Distributed Twitter, the SMS connection.

Also, at BarCampBoston, I am going to try to organize some kind of group to try implementing a Distributed Microblogging application going forward.