I am an obsessive follower of politics. With 189 days left until the US presidential election, I'm in full-on obsessive poll-watching mode.

As usual, you should ignore national polls. The important point is the electoral college. You can see a projected electoral map for 2012, and you can also create your own. My current pessimistic prediction has Barack Obama winning by just 3 electoral votes, hinging on a win in Virginia.

Virginia is one of the four battleground states that are going to matter in this election. The full set, and their current polls, are:

• Virginia - best bet
• Ohio - a possible win
• North Carolina - an unlikely win
• Florida - no chance

Obama won all four in 2008 but this is pretty much guaranteed not to happen this year.

While national polls are almost useless, it is worth keeping an eye on these three:

• Obama's job approval: is he doing a good job? This has been negative for most of Obama's presidency, which is not great but not unusual either.
• Obama's favorability: do you like him? (Whether or not he's doing a good job). Obama's favorability is generally ahead of his job approval. Favorability has typically been a better predictor of electoral success.
• Romney's favorability: another key, because you're unlikely to vote Obama out of office, even if you don't like him, if you dislike the other guy more.

Romney's favorability is currently negative, which is great news for Obama and terrible news for Romney.

This hypothetical conversation never happened; Mr. Santorum's answers are an amalgamation of his public statements on gay marriage and civil unions, and most of the replies are quotes from similar contexts. Sources: [1][2][3]

Okay Twitter, it's time you stopped getting a free pass. You have fucked up your interface, and it's time to fix it.

Before we get to the UI problems, let's reiterate the bigger, older problem, captured just a second ago:

This was cute when you were tiny and still getting over early technical mistakes on the back-end, but you're over that now, you've taken over a billion dollars in funding, you are basically CNN's only news source at this point. You can't be throwing 500s anymore, no matter how cute the whale is. But your problems go much deeper now: even when it renders, your actual user interface is significantly less useful and elegant. Allow me to rant briefly about a few issues. While I'm waiting for the whale to go away so I can take screencaps, here's a shot of the old-old Twitter (via):

Problem 1: right-column layout

The new layout puts tweets down the right, and a mish-mash of useless junk down the left. I know why you did this: you needed to increase the amount of attention paid to promoted followers and trends, because that's your business model. I don't care. You are deliberately distracting your users from something they want to see with something they don't care about. This is the wrong way to do advertising.

Problem 2: tweet composition

Can you tell me the difference between these two ways of composing a tweet? This one is accessible from the left nav:

This one comes up if you click the blue button in the top-right:

The answer is: there is none. They are totally different-looking ways of doing the same thing, both accessible from the top of the front page. Why would you confuse your users like that? If you think the top-right button is too hard to find, why is it there at all? If you think it's useful because it stays visible as the page scrolls, why not make the easy-to-see compose box fixed? Instead you have this weird dual-interface solution that reeks of committees and compromise instead of the great, simple design that you started with.

Problem 3: tweet controls

The basic tweet layout is pretty much unchanged since the beginning:

The friendly "posted X minutes ago" has been replaced by the context-free "27m", but that's a tiny matter. On hover, as before, you get some tweet controls, plus the new "open" link:

And here the real trouble begins. This is what happens when you click "open":

My payoff is that the controls jump to the bottom of the tweet for no clear reason, and I get a more exact timestamp. Is that worth a click? Then why is that link there at all? The answer is because if your tweet has more interactions, like retweets and favourites, you get those here too:

But that's an explanation, not a reason. It would be quite simple to not bother having an "open" link unless there was something interesting to show. But instead we have this weird cruft in the name of consistency. Again, it's a small thing, but all these little things are beginning to add up to a UI that isn't cared about.

Problem 4: conversation view

If a tweet is part of a wider conversation, clicking "open" gives you a lot more context, like so:

Display of larger conversational context is a good idea. But there are two issues: first, the back-end implementation sucks. If I respond twice to your tweet before you reply, that second tweet is lost from the conversation. Sometimes you get the whole conversation, sometimes just the immediately preceding tweet. It's inconsistent and confusing.

And the UI is also inconsistent and confusing: two of the tweets have short timestamps, one has a long. The "hide conversation" close the whole conversation, but uses the same icon as "reply" and is right next to it. In the top-right, where you'd expect the "close" button to be in any other context, there's nothing but a timestamp, unless you hover, where you get this:

So now I have controls for this tweet, and also a "details" link which... closes the conversation, in total defiance of its label. Unless you right-click and open in a new tab, in which case you get the details page for a tweet. Why have a link that only does what it says if you right-click? Why can't I get more details of this tweet inline?

Here's how it should work: I'm looking at a list of tweets, and the bottom one is highlighted. If I want to close the list, there should be a close button in the top-right. If I want details about a tweet other than the last one, I should be able to click it. Is that so hard?

Incidentally, there is a close button for the conversation that's correctly labelled. You get it if you hover over the final tweet:

...right where the "details" link is on the other tweets. And it does the same thing that the already-visible "hide conversation" link did anyway. What on earth is the point?

Problem 5: interactions

This one is so obvious I can't believe it's not been fixed already. This is the default view of the (poorly-named) "@connect" tab:

This view is clumsily mixing together two totally different use-cases. The first is @replies: these are frequent, personal, and demanding of your attention. They are high-value. The second is retweets and favourites: these do not require response (good, because the UI doesn't let you respond anyway), and happen asynchronously: you don't care when a particular retweet or favourite happened to any degree of precision -- so why is it in a timeline? Not that Twitter actually tells you when it happened anyway, since the UI batches up reponses:

It's not like there's not a great, usable UI for handling interactions that already exists to model from: Favstar.fm nailed it years ago. Show tweets in the order they happened, batch up all responses. It's not hard.

And as another tiny little thing: if the "Mentions" link lets you filter down to only @replies, why is there no equivalent "responses" link that lets you filter down to only retweets and favourites?

TL;DR

The New New Twitter has dozens of small, irritating design choices and UI inconsistencies. None of them by themselves is worth a whole post, but together they add up to enough brokenness to complain about. I've tried to keep things constructive by suggesting how they should look instead.

The new look has been out for over a month now and there's been no sign of iteration to fix these things. We wouldn't put up with this crap from Facebook. It's time to fix it.

P.S. Dear commenters, before you immediately point out the many, many UI flaws in this blog, I reiterate that Twitter has a billion dollars, while I maintain this blog in my very rare spare time. I expect more from them, and so should you.

Charles Babbage, one of the father of computers, once wrote:

I will yet venture to predict that a time will arrive, when the accumulating labour which arises from the arithmetical applications of mathematical formulae, acting as a constantly retarding force, shall ultimately impede the useful progress of the science, unless this or some equivalent method is devised for relieving it from the overwhelming incumbrance of numerical detail.

He meant that one day his computer -- at that time seen as a costly and useless device -- would be not just useful but required to make further economic progress. He was right in all but one respect: he thought that day was in the future. In fact, he was already living in it. Scientific and economic progress have always been limited by the available computational power -- but until quite recently, the level of power available never changed, so the limit was imperceptible.

Now that computing power routinely grows by orders of magnitude, it is easier for us to grasp the idea that we are limited by available computation -- it would be nice to decode genes faster, fold proteins quicker, make more accurate weather predictions. We understand these things will get better, in the same way that Babbage could grasp that calculating logarithmic tables (the primary purpose of computation in the 1850s) would someday be quicker.

What is more difficult for us to grasp is that we are still horribly limited by our lack of computational power. We cannot see how limited until those limitations are lifted. Take, as the shadow of an example, the way that one can browse live online maps from one's phone and already scarcely remember how one got by before that ability. In the future, ubiquitous and mind-bendingly powerful computation will make these tasks seem as divorced from utility and convenience as grinding gears to create logarithmic tables seems now.

Human beings are bad at predicting phase changes. We can predict iteration but not invention, so we write stories about giant spaceships that have wired telephone lines. You have no idea how amazing the future is really going to be.

MG Siegler on Google integrating Google+ into Chrome:

Right now, Google bakes G+ into most Google properties via the black nav bar. This undoubtedly spurs a lot of usage. What if the next phase is to take it a step higher? Go right to the browser itself? ... They already have users logging into Chrome now for syncing, etc. We’re already much closer to this happening than most probably realize. ... From a business and integration perspective, these would ... be smart moves. But they’ll also remind people even more of 90s-era Microsoft. Google has to tread carefully here.

I think most people agree that Google should try to avoid being like 90s-era Microsoft, if only because everybody hated 90s-era Microsoft. But there's a bigger reason than Google's long-broken promise of "don't be evil", which is that this strategy doesn't work. You cannot make a bad product popular by integrating it into a good one; all you can do is ruin the good product.

Microsoft, of course, is famous for this. They integrated their awful browser deeply into their operating system. On the face of it this would seem like a vindication of the strategy: Explorer became the dominant browser for years. But while everybody used Explorer, they'd ruined Windows: Explorer was one giant security vulnerability, leading to a huge decline in public confidence in the operating system, making Apple's locked-down approach more attractive. It was a public-relations nightmare, over and over.

And there are lots of much clearer examples. Yahoo's integration of, well, everything into everything else. You name it: they integrated Mail into the front page, News into Mail, YAP into both front page and Mail, My Yahoo into everything. RealNetwork's integration of so much crap into their media player that everyone abandoned them. More recently, Apple's integration of Ping into iTunes was met with derision and further complaints of bloatware. None of these integrations made the crappy products any less crappy, they just made people abandon the popular products in favour of cleaner, simpler alternatives.

Of course, making use of the popularity of your existing, popular products to boost the popularity of your new product is a sensible strategy: you get a burst of traffic and can reach critical mass fast. But if it turns out the new product stinks, you need to turn it off fast. Google tried it with Buzz and it was a disaster; they tried it with Plus and so far it's been working, because Plus is a much better clone of Facebook than Buzz was a clone of Twitter.

But the shine seems to be wearing off Plus, and they're not turning around the APIs they need fast enough to bring developers onto Plus as a platform, a crucial plank of Facebook's popularity. If they build notifications into Chrome to artificially boost its popularity rather than beefing up the APIs to genuinely improve the product, they'll find they have a revolt on their hands, with users abandoning Chrome.

Hopefully, Google knows better than to make that mistake.

Think about a movie that you're downloading via BitTorrent*. Better yet, visualize it using this amazing BitTorrent visualization. Where is the movie stored? Well, on the hard drive of the person who started the torrent. Except they might have left the swarm by now. So instead, on the hard drives of a couple dozen other users. Except none of the ones you're talking to may actually have a full copy of the movie, just the pieces of it you happen to need.

Instead, stop looking at the nodes. They come and go, none of them are vital. Instead, look at the center of the simulation: the mass of bits flying from one node to the next. That's where the movie is. This is the cloud: the real cloud, not the marketing term. The movie isn't on a device, it's on the network, perpetually in transit, stored in the very wires and routers that compose it. And as long as enough nodes exist to bounce it around, it will stay there. Already, there are some torrents that have stayed alive for over seven years, almost from the birth of the protocol. And as the size of the online population increases, the volume of data stored in the real cloud will increase.

Search engines like The Pirate Bay, currently used mostly for purposes of dubious legality, will become the first guides to this new, amorphous universe of disembodied content. Content independent of storage, independent of source, stored in the body of the network itself. No vendor can rent this cloud to you by the hour, nobody can buy it or sell it or control it at all. That's part of its power. That's part of why the real cloud is so very different from the way data has existed before. You don't decide what gets stored on the real cloud: the cloud does, in the ultimate participative democracy. All you can do is feed new things to the cloud and hope it likes it, and provide new tools for getting data in and out.

This is a big deal. It is a phase-change in the nature of data, from liquid to gas. We are only just beginning to see and understand the implications of this transition. Serious research into the real cloud, and tools built around it and on it, are being stunted by the association torrents have with illegal behaviour. But that's going to fade. If you're looking for a blue-ocean field of emerging technology, where you can do new, exciting things and make a big difference, look at the real cloud.

* which you obviously don't do, since you use BitTorrent only to legally download free Linux distributions. So imagine a hypothetical movie.

The only skills gap bigger than the one for programmers is the one for statisticians.

The whole web industry is accumulating vast quantities of data and storing it, magpie-like, as if it has intrinsic value, aided by ever-falling prices for storage. But the data isn't valuable. It doesn't mean anything until somebody who knows what they're doing looks at it, sifts through it, and produces a tool that lets others use it to draw valid and useful conclusions.

But hardly anybody does this. Instead we apply the most absurdly basic analyses and build whole businesses around them. We are messing around in the shallows, while the ocean of data gets bigger every day.

If you want to find yourself enormously over-employed for the next decade, learn a bunch of statistics. As a bonus, find a way to fit machine learning in there, but we even have way more people who understand machine learning than understand what it is we should be teaching them.

I tweeted about ORM last week, and since then several people have asked me to clarify what I meant. I have actually previously written about ORM, but it was in the context of a larger discussion about SQL and I shouldn't have confused the two issues. So here I'm going to focus on ORM itself. I'm also going to try to be very brief, since it became very apparent from my SQL article that people tend to stop reading at the first sentence that makes them angry (and then leave a comment about it, whether or not their point is addressed later on).

What's an anti-pattern?

I was pleased to discover that Wikipedia has a comprehensive list of anti-patterns, both from within the world of programming and outside of it. The reason I call ORM an anti-pattern is because it matches the two criteria the author of AntiPatterns used to distinguish anti-patterns from mere bad habits, specifically:

1. It initially appears to be beneficial, but in the long term has more bad consequences than good ones
2. An alternative solution exists that is proven and repeatable

It is the first characteristic that has led to ORM's maddening (to me) popularity: it seems like a good idea at first, and by the time the problems become apparent, it's too late to switch away.

What do you mean by ORM?

The chief offender that I'm talking about is ActiveRecord, made famous by Ruby on Rails and ported to half a dozen languages since then. However, the same criticisms largely apply to other ORM layers like Hibernate in Java and Doctrine in PHP.

The benefits of ORM

• Simplicity: some ORM layers will tell you that they "eliminate the need for SQL". This is a promise I have yet to see delivered. Others will more realistically claim that they reduce the need to write SQL but allow you to use it when you need it. For simple models, and early in a project, this is definitely a benefit: you will get up and running faster with ORM, no doubt about it. However, you will be running in the wrong direction.
• Code generation: eliminating user-level code from the model through ORM opens the way for code generation, the "scaffolding" pattern which can give you a functional interface to all your tables through a simple description of your schema. Even more magically, you can change your schema description and re-generate the code, eliminating CRUD. Again, this definitely works initially.
• Efficiency is "good enough": none of the ORM layers I've seen claim efficiency gains. They are all fairly explicit that you are making a sacrifice of efficiency for code agility. If things get slow, you can always override your ORM methods with more efficient hand-coded SQL. Right?

The problems with ORM

Inadequate abstraction

The most obvious problem with ORM as an abstraction is that it does not adequately abstract away the implementation details. The documentation of all the major ORM libraries is rife with references to SQL concepts. Some introduce them without indicating their equivalents in SQL, while others treat the library as merely a set of procedural functions for generating SQL.

The whole point of an abstraction is that it is supposed to simplify. An abstraction of SQL that requires you to understand SQL anyway is doubling the amount you need to learn: first you need to learn what the SQL you're trying to run is, then you have to learn the API to get your ORM to write it for you. In Hibernate, to perform complicated SQL you actually have to learn a third language, HQL, which is maddeningly almost-but-not-quite SQL, which then gets translated to SQL for you.

A defender of ORM will say that this is not true of every project, that not everyone needs to do complicated joins, that ORM is an "80/20" solution, where 80% of users need only 20% of the features of SQL, and that ORM can handle those. All I can say is that in my fifteen years of developing database-backed web applications that has not been true for me. Only at the very beginning of a project can you get away with no joins or naive joins. After that, you need to tune and consolidate queries. Even if 80% of users need only 30% of the features of SQL, then 100% of users have to break your abstraction to get the job done.

Incorrect abstraction

If your project really does not need any relational data features, then ORM will work perfectly for you, but then you have a different problem: you're using the wrong datastore. The overhead of a relational datastore is enormous; this is a large part of why NoSQL data stores are so much faster. If your data is relational, however, that overhead is worth it: your database does not merely store your data, it represents your data and can answer questions about it on the basis of the relations captured, far more efficiently than you could in procedural code.

But if your data is not relational, then you are adding a huge and unnecessary overhead by using SQL in the first place and then compounding the problem by adding a further abstraction layer on top of that.

On the the other hand, if your data is relational, then your object mapping will eventually break down. SQL is about relational algebra: the output of SQL is not an object but an answer to a question. If your object "is" an instance of X and "has" a number of Y, and each of Y "belongs to" a Z, what is the correct representation in memory of your object? Is it merely the properties of X, or should it include all the Ys, and/or all the Zs? If you get only the properties of X, when do you run the query to fetch the Ys? And do you want one or all of them? In reality, it depends: that's what I mean when I say SQL is the answer to a question. The representation of your object in memory depends what you intend to do with it, and context-sensitive representation is not a feature of OO design. Relations are not objects; objects are not relations.

Death by a thousand queries

This leads naturally to another problem of ORM: inefficiency. When you fetch an object, which of its properties (columns in the table) do you need? ORM can't know, so it gets all of them (or it requires you to say, breaking the abstraction). Initially this is not a problem, but when you are fetching a thousand records at a time, fetching 30 columns when you only need 3 becomes a pernicious source of inefficiency. Many ORM layers are also notably bad at deducing joins, and will fall back to dozens of individual queries for related objects. As I mentioned earlier, many ORM layers explicitly state that efficiency is being sacrificed, and some provide a mechanism to tune troublesome queries. The problem, I have discovered with experience, is that there is seldom a single "magic bullet" query that needs to be optimized: the death of database-backed applications is not the efficiency of any one query, but the number of queries. ORM's lack of context-sensitivity means that it cannot consolidate queries, and must fall back on caching and other mechanisms to attempt to compensate.

What are the alternatives?

Hopefully by this point I've made some kind of case that ORM has fundamental design flaws. But to be an antipattern, there needs to be an alternative. In fact, there are two:

Use objects

If your data is objects, stop using a relational database. The programming world is currently awash with key-value stores that will allow you to hold elegant, self-contained data structures in huge quantities and access them at lightning speed. There's no law that says Step One of writing any web app is installing MySQL. The massive over-application of relational databases to every data representation problem is one of the reasons SQL has acquired a bad reputation in recent years, when in fact the problem is lazy design.

Use SQL in the Model

It's hugely dangerous to claim there is One True Way™ to do anything in programming. But in my experience, the best way to represent relational data in object-oriented code is still through a model layer: encapsulation of your data representation into a single area of your code is fundamentally a good idea. However, remember that the job of your model layer is not to represent objects but to answer questions. Provide an API that answers the questions your application has, as simply and efficiently as possible. Sometimes these answers will be painfully specific, in a way that seems "wrong" to even a seasoned OO developer, but with experience you will get better at finding points of commonality that allow you to refactor multiple query methods into one.

Likewise, sometimes the output will be a single object X, which is easy to represent. But sometimes the output will be a grid of aggregate data, or a single integer count. Resist the temptation to wrap these in too many layers of abstraction, and deal with the data on its own terms. Above all resist the fallacy of OO, that it can represent anything and everything. OO is itself an abstraction, a beautiful and hugely flexible one, but relational data is one of its boundaries, and pretending objects can do something they can't is the fundamental, root problem in all ORM.

In summary (TL;DR)

• ORM is initially simpler to understand and faster to write than SQL-based model code
• Its efficiency in the early stages of any project is adequate
• Unfortunately, these advantages disappear as the project increases in complexity: the abstraction breaks down, forcing the dev to use and understand SQL
• Entirely anecdotally, I claim that the abstraction of ORM breaks down not for 20% of projects, but close to 100% of them.
• Objects are not an adequate way of expressing the results of relational queries.
• The inadequacy of the mapping of queries to objects leads to a fundamental inefficiency in ORM-backed applications that is pervasive, distributed, and therefore not easily fixed without abandoning ORM entirely.
• Instead of using relational stores and ORM for everything, think more carefully about your design
• If your data is object in nature, then use object stores ("NoSQL"). They'll be much faster than a relational database.
• If your data is relational in nature, the overhead of a relational database is worth it.
• Encapsulate your relational queries into a Model layer, but design your API to serve the specific data needs of your application; resist the temptation to generalize too far.
• OO design cannot represent relational data in an efficient way; this is a fundamental limitation of OO design that ORM cannot fix.

You may have heard about Pete Warden's iPhoneTracker, an app that lets you explore the giant trove of geolocation data your iPhone has been collecting since iOS 4.0 (and possibly before).

You may not know that the grid on Pete's released app is the result of his app deliberately aggregating the datapoints to a grid, in order to be a little less creepy:

if you zoom in you’ll see the points are constrained to a grid, so your exact location is not revealed. The underlying database has no such constraints, unfortunately.

But hey, why should he decide how much we want to expose our location? Let's get super creepy! Following some instructions from a clever friend, I made the very simple change required to increase the granularity of the data shown on the map. Before:

Woah! Neat, right? If you want to try it out yourself, you can follow Nicole's instructions on your own downloaded copy of Pete's source from github, or if that's too much trouble and you trust me, you can download your own copy of the extra-creepy version of iPhoneTracker. I'm not very experienced with compiling desktop software, but this works for me, and I use OS X Snow Leopard, so it will probably work for Leopard OS X too.

Important note: "granularity" is not the same as "accuracy". Your iPhone is frequently wrong about where you are, by up to half a mile or so. So your data points will show on average about where you were, but there will be plenty of random outliers -- which is why I appear to spend so much time swimming in San Francisco Bay, for example.

Enjoy!

When you say "agile", I hear "cargo cult".

Agile is a process for managing software development. If you have a great team of smart people who communicate well and trust each other, they can use agile techniques to release lots of small iterations on a software project very quickly. This pattern of software release is often useful for startups. None of this is in dispute.

The problem is that with its rise in popularity, it has been both misunderstood and over-applied. If you have a good software team you can use agile, but if you use agile you will not automatically get a great team. If your team members communicate well and trust each other they can use agile, but if they communicate well and trust each other they could use any other methodology up to and including no fixed process whatsoever, and be equally successful. Agile changes your release pattern, not your people.

Bottom line: great teams produce great software. Great teams using agile release software every two weeks. Bad teams will produce shitty software. Bad teams using agile will release shitty software every two weeks.