God is broken. Please try again later.

posted 23 October 2003

There is growing speculation around the web that Google is broken, including from the most authoritative of sources. The evidence is all over the place, but Google remains tight-lipped about the problem, possibly in light of an upcoming IPO but more likely because they simply don't want to admit that the single most-used technology service in the world is broken.

So what's broken, and how is it broken? Well, there are now loads of discovered search word combinations that produce big counts of results, but don't actually list them (first discovered by eager GoogleWhackers). This seems to have been a poorly-designed patch introduced to combat people who are creating link farms and otherwise trying to cheat the index: by blocking them, it also blocked everything useful.

However, more serious problems lie underneath. The evidence indicates that Google has a theoretical size limit of 4.1 billion pages because of the way it indexes pages, and that efforts to keep results relevant (a "freshbot") has reserved 500,000 of these index numbers, lowering that limit to 3.6 billion -- a number, based on previous growth trends, that Google should be hitting right around now. Evidence is mounting that Google is producing workarounds to this problem rather than go through the painful process of re-coding from their entire code base (which is necessary in the case of such a fundamental limit), producing a "supplemental" index, although it may be that the supplemental index is simply a patch until they finish the re-coding, which it's been speculated could take up to a year to complete. In the meantime, even moderately heavy users like myself have noticed that Google's results have been bizarre and unreliable recently, a lot like Yahoo! and Infoseek results in the bad old days of search.

All those links above are full of people speculating on the technological side of things, but what about the user side? Google has become fundamental to the way people use the web: people have become used to the power of an engine that has almost single-handedly solved the problem of universal information access -- it's quite hard to think of things that Google can't tell you something about. So what happens if it turns out Google doesn't scale? Having become used to the godlike power of Google, what will we do without it?

What do you do when god is broken?

tagged with