Scaling a Microblogging Service – Part II

When discussing microblogging scalability, the conversation includes scaling each individual service, but also scaling the network and relationship between services. Part I discussed the challenges of scaling a single microblogging site with focus on dealing with a large and constantly changing content database. In that post I mentioned that the proposal by some critics to build a distributed or federated microblogging service as a scaling solution will actually make thing worse. This second part will elaborate on that claim.

When discussing a distributed microblogging service, the conversation touches the long debate on the future of social networks and linking communities across individual walled gardens. After all, microblogging is one aspect of the social web, and status updates lives side by side sharing photos, videos, and other personal information and experiences. Being able to choose a social network and make friends from another without having to sign up for multiple accounts is one of the visions being offered. Another is the approach being advocated by the Data Portability group, which focuses on being able to move an entire experience off to another network instead, creating multiple identities.

There are many ways in which a distributed microblogging service can be architected, but the core functionality is pretty consistent. Users should be able to view status messages from friends on other social networks without having to join and move around between them. While this would be cool to have for photos, it is much more relevant for microblogging content. At the heart of it, microblogging creates a conversation that lives somewhere between email and instant messaging, and there is great value in being able to extend the conversation to where the right audience is, not just the local community one has happened to belong to.

As a user of a new microblogging service, I would like instead of just saying ‘follow Joe’, to say ‘follow Joe on Twitter, Sue on Pownce, and Steve on Jaiku’. And be able to do all this from my own private microblog which can live right next to my own WordPress installation (or any other blogging platform). This is part of the vision offered by the DiSo community. As long as my own service is up, I can send updates, and if any of the other services is down, they only impact a small subset of my friends (assuming they are not all using a single service). It also allows my friend to host their own presence and still communicate with me.

The claim that a distribute solution by itself will solve reliability is false. Taking a lesson from email, the idea that if one service is down people can simply hop over to another is just not practical. If your email provider is down you can easily switch to another, but that only solves the sending problem. All your friends are still trying to reach you at your old address. Same with microblogging, if Twitter is down – switching to Pownce doesn’t really solve the problem. What does solve it is a redundancy solution at the Twitter and Pownce level.

Getting any individual microblog to be scalable and stable is a challenge all by itself. As long as the individual pieces are having problems, a network of multiple sites is only as stable as its weakest link. Distributing microblogging across multiple services does not resolve the stability problems. It does offer more options and helps people move around between services, and will eventually lead to better services as they try to compete. But that is a free-market solution, not a technology solution.

Attempting to link multiple sites together before each one is capable of handling a significant load is only going to make things worse. To understand this claim, we need to examine how such a federated network would work. In the following scenario, Moe and Larry are Twitter users, while Curly uses Pownce. Moe would like to follow both Larry locally and Curly via a federated connection. When requesting to follow each friend, Twitter will check if their profile is public or private. If private it will request permission to from the friend. This is simple for Larry as he is a local user and Twitter has full control end-to-end. However when asking permission to follow Curly, Twitter has to negotiate this with Pownce. For the scope of this discussion we’ll assume all profiles are public. When Moe visits his Twitter home page or a desktop application, he can see the latest status updates from both Larry and Curly seamlessly.

For a Twitter user to follow the updates of a Pownce user, two things need to happen. First, Twitter needs a way to know where to find the remote user and how to request his updates on Pownce. This is can done by building custom bridges between the two services, or using a microblogging interoperability protocol that will be created. Second, Twitter needs to fetch the Pownce updates and present them within the Twitter application. For this to work – either Twitter subscribes to a Pownce feed (such as XMPP) and continuously receive streaming updates for every Pownce user every Twitter user wants to follow, or it grabs the relevant updates from Pownce as needed when Twitter users visit the site (or make API calls). In other words, Twitter can start replicating all of Curly’s messages locally, or request them every time Moe checks his friends’ timeline.

As mentioned in the part I, there are two subsystems inside every microblogging service – delivery and retrieval. No matter which approach is used (updates subscription or updates requests on demand), someone has to deal with the complexity of scaling the retrieval system. If Twitter is subscribing to Pownce’s XMPP feed, it makes Pownce’s work much easier as the overhead of pushing out messages in real-time over an XMPP Pub/Sub implementation is generally simple and low on resources. But it means Twitter has to store all those updates under virtual accounts since those users are not local to Twitter. When the Twitter user uses the website to follow his Pownce friends, Twitter has to retrieve those messages just like any local user. If Twitter opts to fetch updates as needed, Pownce has to retrieve the messages.

Anyway you look at it, someone has to deal with getting those messages from the remote system no matter where they are stored. Assuming a balanced network of connected sites, eventually everyone will need to deal with scale. But while some sites are still not facing the same level of load Twitter is facing with millions of subscriptions, if they started creating local store of Twitter updates, they will find themselves in the same boat without any actual user growth.

The idea of duplicating content across sites is as unappealing as the idea of scaling a single site using the inbox solution (creating a copy of each update per follower). If this is what we are building, a solution where people subscribe to content that is delivered to their microblogging inbox, we might as well just use call it by its common name: email mailing-lists. Duplicating also makes deleting messages practically impossible, something most users expect. And of course, any smarter subscription across services based on metadata will require complexity and load that will rule out the idea of personal microblogging services.

It is somewhat surprising that in a service space that is all about live updates from a wide range of devices – many of which are mobile – the scaling challenge both locally and distributed is in data lookup and not distribution. The unique nature of microblogging has the characteristics of both email and instant messaging where immediacy is combined with persistency and loosely coupled groups.

This series is concluded in part III about the challenges in designing scalable microblogging APIs, and how API decisions play a significant role in performance issues.

5 thoughts on “Scaling a Microblogging Service – Part II

  1. Hi Eran,
    I stumbled across you’re blog through the oAuth notification and then browsed through some of your older articles (like this one).
    I’m loving the pieces you’ve posted – they are much more concise, indepth and knowledgable than anything I find on the bigger blogs.
    Also, most of the topics you are talking about are close to my heart. We’re busy with a tool/service that would aim to tie together your profile, activity, etc and broker that information with 3rd party’s through a trusted gateway (we’re very excited about the new oAuth draft released!).
    I’d love to bounce some ideas off you and see where we can perhaps share thinking and get dataportability going. You can contact me at the email supplied in the comments.

  2. You are kind of describing a DNS for microblogging.
    Imagine a world where tweets, or status updates are like dns updates, broadcasted and infinitely mirrored across the entire social network. At once, they exist on local services such as your private blog, and also accessible and backed up by the high availability of services like facebook or google, via a protocol that everyone is free to implement and guaranteed to be understandable decades or even centuries into the future
    I don’t see the scalability of messaging services like twitter to be insurmountable. The technical community has designed systems that have handled much more data with a much higher degree of complexity than twitter. Of course, it might take a few hundreds of millions of dollars to get there.

  3. Why not do the reverse of “inbox”, which is “outbox”? Every user has a JSON page that lists their last N messages, stored as a plain text file and served by a plain webserver.
    When someone views their twitter page, it pulls the outboxes of their friends (via plain old script calls or AJAX), then weaves the results together. (It could animate the page if requests come in late).
    The browser+server will use e-tags to minimize the bandwidth. Adding proxies/caches will help. Sharding is trivial. There is an incentive to follow fewer people because you page loads faster. Cross-domain services are trivial.
    Am I missing something?

  4. Why doesn’t the distributed status update fit the same pattern as weblog syndication? We have a mechanism for following the distributed updates of many authors already, and it is RSS.
    If I create a Google Reader account and subscribe to Joe’s, Sue’s, and Steve’s RSS/Atom feeds, I’ll have a great prototype of a distributed/federated system for following status updates.
    A standard news reader may not be exactly the right tool, since they use aggressive caching and assume that each post has a distinct title and body. But if a clever developer set out to build a reader specifically for social-networky status updates, she could easily do so, without ever touching a custom protocol or api.
    It’s a duplication approach, but it distributes storage and aggregation overhead to the client. Since all the posts and subscriptions are linked, you can easily get to the friend-of-a-friend features offered by the home social networks.
    I say let a million Twitters bloom, and let’s just build some good “friend-following” newsreader apps.

Comments are closed.