This week, I discovered that she’s now on Twitter as @Natalie_Gulbis. She is, as far as I know, the first top-tier professional golfer of any tour to be on Twitter. Yay, Natalie!! That she would be the first is totally in keeping with her accessible, outgoing and friendly personality.

At this writing (Dec 29. 2008), she only follows 4 people and is followed by only 234. But I give her and her agent major props for realizing Twitter is a good place to be. Like anyone new to Twitter, it may take a little time to discover how it works. I’m hoping that unlike many celebrities, she will really get in and talk with people.

Of course, someone like Natalie Gulbis needs to be careful with the public. I can imagine she has more than her share of unwanted attention. But at least on Twitter, you don’t have to hear from anyone you don’t want to. And there is so much to be gained from engaging with people and making new acquaintances and friends, not to mention drawing non-golfers into the sport.

If I may make a suggestion, it will be the same as to anyone new to Twitter. Listen first, follow first. If you hear something interesting chime in. Make an observation, a joke, ask a question or state your opinion. Most importantly, be a person, not an inaccessible celebrity. Twitter is a conversational medium, a means for introducing and connecting people, not just another broadcast channel. Let that sunny personality of yours show through. You’ll find we’re a friendly lot and you’re more than welcome to join the conversation.

So I’ve been putting together the following list of what I consider to be important features that a good microblogging function should have. In most cases, I’ve approached it from how it should appear to the user. If you are coming to DevHouseBoston on Sunday, you might want to read this list over and come prepared to argue the merits of different points, add your own, or if you’re a UI designer, contribute to a discussion of how to design a UI that incorporates these features.

Remember that one of the benefits of an open system is that there can be many different UIs to choose from!

So, following, is the list in a very raw and unpolished form.

Note: DMB means Distributed Microblogging.

Feature list for dmb UI

Omnipresent replies, direct messages tabs or buttons.
Tags on other microblogs, on posts, on other tags?

Direct support for conversations, replies, etc.
Main page. Like twitter. Update input box at top, but maybe with separate scrolling list of stream.
Multiple streams showing, replies, all, selected person’s archive, including user’s archive.
“Hide” check boxes for each user. Clicking it removes their updates from stream list, and places their name in the a “hidden” list for easy reinstatement.

Also would be able to show/hide certain groups.

Conversation support. Each tweet would have a “reply” button. Clicking it would add the @user_name to the beginning of the update, and also make a database entry linking the update to one or more previous updates. The @’d user names would be like the “to” list on an email. Each update could have multiple other updates it was in-reply-to. This is a many-to-many relationship. An update can be in response to multiple other updates, and clearly an update can have many responses to it.

Updates have GUIDs. Required by IMPP.

Tabs for different subsets of users = from Mike Gaines. “tag-tab”

Separate input area for different tag-tabs.

Topics for User experience
Conversations. What can you do? How are they shown?
Multiple addressees (like email?)
Latest on top or on the bottom? User pref?
Classes of messages
Direct
Narrowcast (to a group tag?)
Broadcast (or is this simply to all tags?)
Temporary silencing (show/hide checkbox) So I can turn off someone who is usually ok, but is just ranting or something.

Ad-hoc grouping. Drag users into a tab where you see only their tweets.
Public ad-hoc groups. All your followers can see tweets. You just filter locally.
Private ad-hoc groups. Just the people in the group see the tweets.

General question.. will adding private groups destroy some of the vibrancy and serendipity of Twitter?

How to present tagging in the UI so it’s intuitive with what it does for the user, as opposed to a geeky abstract function?

What can be tagged, and hence grouped? Users, convos, even other tags? Locations? languages? specific tweets having to do with a certain subject? others?

Universally unique tags vs. private tags. UUtags would be very helpful for organizing large tagging efforts. But, how to agree on a tag for say, “SocialMediaBreakfast8″?

theRQ would be great to be able to send/post an out “out of stream” message to someone that they get upon login- wouldn’t age off. (subject to spam abuse?)

Search – Let Google do it? What about private mblogs? Can individual servers be expected to support a general search capability? Thru XMPP? Thru HTTP?

Show me when I have new, unread DMs.

Multiple addressees for DMs. Now it’s beginning to look like email.

Show me when I have new, unread messages.
Show me when I have new, unread messages from specific people or groups. (on that tag-tab?)

Auto-update (ala chat) vs. manual update.

Rich content. Photos, audio. (How to transmit to mobile devices?)

How to add these features without sacrificing the simplicity?

A few lines when you request to follow, so you can explain why. Either that or a listing of common friends on different sites.

What about the Twitter community? Can it be made part of DMB? Can we build an interface to Twitter? Will Twitter participate in DMB?

What about other communities that start up? Twitter has very amorphous sub-communities.

Implementation questions:

Use XMPP archiving or external? (some xmpp archiving options not applicable, like off the record, although they might be of interest)

How to do multiple servers?

How to map mb posts onto “collections” in xmpp archiving. Another reason to use external archving?

If we did external archiving it would relieve the burden of archiving on servers that could be used. Although they’re more likely to support archiving than pubsub.

I had the pleasure of speaking at BarCampBoston3 this past Saturday in Cambridge, MA on the topic “Distributed Twitter”. This was a well-attended event and the geek energy was high! BarCamps attract and cater to hard-core coders and implementers. Not many marketing, PR or media types were in evidence, but there were quite a few startup CEOs there looking for help.

One particularly useful session for me was at the start of the day, when Matt Douglas, CEO of MyPunchBowl.com, brought together companies looking for help with developers looking for work. It was very informal, we all got 10 seconds to say who we were and what we were looking for. I made a couple of promising connections there, so that in itself was worth the 90 minute drive from my home in Connecticut to Cambridge. Thanks, Matt and MyPunchBowl! This kind of Hook-me-up session should be a part of every BarCamp.

I was pleased that my session on Distributed Twitter attracted a packed room. Unfortunately, I completely zoned and missed the fact that sessions were only 1/2 hour, so I had to cut things short, but I did meet some interesting people who were interested in trying to get a project off the ground to actually build something.

I had some interesting follow-up discussions later in the day with people who were at my talk, but unfortunately, did not get to tell all at the meeting about the Distributed Twitter Google Group. Probably no biggie since I’m going to have to change the name of the group anyway, to avoid the inevitable cease-and-desist letter from Twitter.com’s lawyers for using their name.

One of the big reasons I went to BarCamp was to try to connect with mobile developers since a big part of decentralizing a microblogging platform is to provide the mobile connection that Twitter has. Many people use Twitter almost exclusively on their phones. I did meet two young fellows who seemed very interested in creating a mobile app to interface directly to a microblogging platform. Their advice was to skip SMS for demo purposes and simply use the mobile IP connectivity. Their POV was that SMS will go away in due time to be replaced by the more flexible and extensible IP protocol directly to apps on the phone. I’m not sure I buy the argument that you don’t need to have an SMS connection. SMS will be around for a while to come yet, and not having it could hamper uptake severely.

I had other interesting conversations regarding the use of XMPP as the “carrier” protocol for microblogging services. It was suggested to use Google’s gTalk service to experiment with since it’s a fully compliant (I think) XMPP end-point.

I continue to have some challenges with XMPP proponents not fully appreciating that microblogging is more like blogging than instant-messaging or chat. I’ll confess I don’t understand the differences completely myself, but I do see that there are some and that they are significant to a proper implementation. For instance, microblogging does not have the requirement or notion of presence. As a user, you don’t need to be “present” to get all the updates that people may send. You should be able to go back at any time and get the complete history of a microblog’s updates, just like it was a blog. Related to presence is the notion that senders of content aren’t responsible for keeping a permanent copy around for receivers to reference and search. More about this at a later time, but one shouldn’t have to rely on being “present” or having your receiving server make a copy of everything that a microblog sends to have a complete record. The sender should be responsible for keeping a permanent, permalinkable copy.

So, all in all, BarCampBoston3 was definitely worthwhile but I could have done better had I taken two consecutive sessions, which is allowed, and not have been so pressed for time.

Coming up…

In the weeks to come, I’m scheduled to do two more presentations about Distributed Microblogging.

1. IgniteBoston3 - May 29th, 6 pm at Tommy Doyle’s in Harvard Square, Cambridge, MA. This will be a quick, 5 minute “drive-by” preso just to introduce people to the topic. I’ll probably just get enough time to describe what it is, why I’m involved and to pimp preso #2 (next) and refer people to the Google group. If you’re interested in coming, be sure to register because they always fill up!
2. Berkman Center Thursday Night Meeting - June 5th, (time: tba), at the Berkman Center for Internet and Society, Harvard University, 23 Everett St. Cambridge, MA. I’ll be speaking on Distributed Microblogging. I’m not sure, but there may be other presenters as well. This is my first time at a Berkman Thursday Night meeting and I’m very excited.

Ok, so let’s talk about the hard bits of doing Distributed Microblogging. It’s easy to envision a multitude of servers exchanging microblog posts, and a UI that simply arranges the posts in chronological order. By the way, chronological order is easy to do if all the servers are synched with a time standard like nist.gov, and most are. But the hard bit is making it work in a way that performs well and scales to large populations of both microblog posters and readers. I’ve been thinking of some different alternatives for this, which I’ll lay out here. As always, your thoughts are welcome.

Performance

So, what do I mean by “performs well”? Well, microblog (ie, Twitter) updates happen much more frequently than what we’d consider traditional blog posts, but not quite as fast as instant messenger or chat updates. And microblogs don’t have the notion of presence attached to them. You don’t worry about whether a Twitter poster is “on line” at any particular time, although you may deduce that from the rapidity of their responses to you.

To put a number on it, I’d say that a microblog post notification should be transmitted in less than a minute, ideally in a few seconds, although a delay of up to 5 minutes is not that objectionable. I frequently will ignore my Twitter feed for many minutes, sometimes hours. I have no expectation that I will see someone’s posts immediately (i.e., less than a second) nor do I care to. Microblogging is a river of updates. You don’t expect to see every single one.

Size

Now let’s talk numbers of followers and following. A typical Twitter user has 100 or fewer people that they follow, and a similar number that are following them. But the edge cases are far bigger. Someone like Robert Scoble (@scobleizer) or Chris Brogan (@ChrisBrogan) follow literally thousands of other users, and have even more following them.

Ok, so we need to be able to update thousands of users in nominally less than a minute, but at the exreme less than five minutes. If you don’t buy that, please leave a comment explaining why these limits are not realistic.

Alternatives

RSS feed polling

The simplest solution would be polling, which is reader initiated. Followers would poll the feeds of the microblogs they were interested in. It seems to me that polling has to be discarded as a solution except for occasional or retrospective uses. I think a solution should include RSS feeds but it seems obvious to me that for someone with 100 friends to poll those RSS feeds every minute, or ideally faster because they might post something is a stupendous waste of bandwidth.

On the sending side, if that same person had 100 followers, each of those followers would be polling the person’s RSS feed every few seconds causing a very high server load. Now multiply that by however many people’s accounts are hosted on that server and the problem quickly blows up.

Notification

A much more efficient solution would be for senders (microblog posters) to notify followers when a new post has been made, and perhaps to proactively send the new content in the notification message. Then, resources are used only when there is actual traffic to send. Both senders and receivers are quiescent when no one is posting. So what are the possibilities for implementing notification? I see the following.

1. RSS cloud API – a little-known part of the RSS specification, the cloud element allows a feed to publish a web-service address that readers of the feed can register with to be notified of changes using a SOAP or xml-rpc call.
2. Jabber (XMPP) channels between DMB servers to carry notifications and content.
3. UDP notification with http callback. UDP is lightweight for both senders and receivers. No open connections are required between senders and receivers. It’s sort of like RSS cloud, but narrowly and specifically designed for DMB, as opposed to generalized RSS.

RSS cloud API

The cloud API was specifically designed with this purpose of notifying readers of content updates. Its original intent, judging from the RSS 2.0 spec was to allow actual client feed readers to register with the cloud. In the case of DMB, it would be cooperating servers that would register for notification with each other.

The problem with RSS cloud is overhead. Microblog entries are tiny and frequent compared with blog entries or traditional site updates. To require a follower server to read an entire RSS document to get 140 characters of content, and have this happen every few minutes when the poster updates would be inefficient to say the least. In addition, there is experience with the cloud api that indicates just the HTTP session overhead for notifying many users becomes intolerable, although this was from the perspective of actual clients being notified as opposed to clients’ servers being notified.

Jabber (XMPP)

Jabber is a very tempting candidate for this application, and has been getting quite a bit of discussion in the development community of late. Here’s an example. The advantage to Jabber is that it maintains open sessions between servers, which eliminates the session setup/teardown overhead, and allows for almost instantaneous notification of all “following” parties.

But this may also be a disadvantage in situations where there are hundreds or thousands of “followers” for a single sender. IM or chat is typically one-to-one, or one to a few, but microblogging is frequently one to hundreds or one to thousands. I am not familiar enough with Jabber servers in actual practice to know what their performance or connection limitations are. Anyone with Jabber implementation or operational experience is strongly encouraged to comment.

Jabber’s concept of presence could be used to keep the number of messages by only requesting messages updates when a user is actually logged into his/her microblog system. What’s interesting about this notion is what “logged in” means. Microblogging, at least the way Twitter works, does not really require the concept of presence. For instance, you can be “logged in” to Twitter, but dormant for hours not getting any updates until you request a page refresh.

In fact, one key aspect of microblogs that differentiates them from IM or chat is that they don’t typically “auto-update”. And rather than this being a disadvantage, I-and I think many others-find this on-demand update to be much more useful than a streaming IM or chat window. It’s really more like reading small blog posts. I read when I want to, not when someone else decides to say something. So, using the presence capability without auto-updating will require a little clever UI design to, in a sense, auto-logoff the user when their web page hasn’t been refreshed in a certain period of time. This then, will also require the ability of followers to query the senders they follow for “back posts”, so they can see what happened in the past without keeping a client logged in all the time to save all the posts.

UDP for notification

In the past year, I did some work on a revamped email protocol I called IMTP that uses small UDP messages to notify receiving servers that the sender has some traffic for them. The receiver then calls back to the sender using TCP to get the message body. This was based on Prof. Daniel Bernstein’s Internet Mail 2000 proposal several years ago. He proposed, quite rightly in my opinion, that mail senders should bear the burden of storing the message contents, not the receivers, and that mail content should only be sent when a receiver actually wants to read it.

The advantage to UDP is that it is very light weight for both the sender and the receiver, not requiring any session overhead or setup/teardown. If used for microblogging, UDP notification messages could take the place of the continuously open TCP sessions that Jabber employs, thus reducing the session resource allocation load on both ends.

Now, of course, the issue with UDP is that it is not guaranteed to be delivered. The IMTP service we built would retry the UDP message at some frequency until the receiver called back to either fetch or reject the message.

Conceptually, using UDP messages with a 140 character payload and a message number would fit a microblogging application very well. Because microblogging has no expectation or requirement of presence or real-time delivery like chat and IM do, a dropped UDP message is not a tragedy. If the sender retried even once or twice per minute, that would be plenty of timeliness for microblogging. Plus, if the UDP messages carry a monotonically increasing message number, the receiver can know if they’re missed a message and simply call back to the sender to get it. The microblogging UI can reconstruct the sequence easily when the missing pieces, if any, finally come through.

A notification system using UDP would seem to minimize the resource requirements on both senders and receivers, and can perform the same kind of message fan-in/fan-out optimization that Jabber could. In other words, a given microblog update body needs to be sent only once to any server, regardless of how many followers there may be on that destination server.

Comments, please!

I am very interested in what others think and have to say about these issues. This problem of efficient and timely-enough notification, it seems to me, is the tough nut to crack for a good solution to microblogging.

By the way, I am going to do a session on Distributed Twitter (or Microblogging) at BarCampBoston3 on May 17, 18. We are going to try to bring in some mobile technology folks to discuss the other really interesting issue with Distributed Twitter, the SMS connection.

Also, at BarCampBoston, I am going to try to organize some kind of group to try implementing a Distributed Microblogging application going forward.