Turn the music up 'til the windows start to shatter.



Databases: how they work, and a brief history

posted 2 days ago

My twitter-friend Simon had a simple question that contained much complexity: how do databases work?

I don't have a job at the moment, and I really love databases and also teaching things to web developers, so this was a perfect storm for me:

The result was an absurdly long thread of 70+ tweets, in which I expounded on the workings and history of databases as used by modern web developers, and Simon chimed in on each tweet with further questions and requests for clarification. The result of this collaboration was a super fun tiny explanation of databases which many people said they liked, so here it is, lightly edited for clarity.

What is a database?

Let's start at the very most basic thing, the words we're using: a "database" literally just means "a structured collection of data". Almost anything meets this definition – an object in memory, an XML file, a list in HTML. It's super broad, so we call some radically different things "databases".

The thing people use all the time is, formally, a Database Management System, abbreviated to DBMS. This is a piece of software that handles access to the pile of data. Technically one DBMS can manage multiple databases (MySQL and postgres both do this) but often a DBMS will have just one database in it.

Because it's so frequent that the DBMS has one DB in it we often call a DBMS a "database". So part of the confusion around databases for people new to them is because we call so many things the same word! But it doesn't really matter, you can call an DBMS a "database" and everyone will know what you mean. MySQL, Redis, Postgres, RedShift, Oracle etc. are all DBMS.

So now we have a mental model of a "database", really a DBMS: it is a piece of software that manages access to a pile of structured data for you. DBMSes are often written in C or C++, but it can be any programming language; there are databases written in Erlang and JavaScript. One of the key differences between DBMSes is how they structure the data.

Relational databases

Relational databases, also called RDBMS, model data as a table, like you'd see in a spreadsheet. On disk this can be as simple as comma-separated values: one row per line, commas between columns, e.g. a classic example is a table of fruits:


The DBMS knows the first column is the name, the second is the number of fruits, the third is the price. Sometimes it will store that information in a different database! Sometimes the metadata about what the columns are will be in the database file itself. Because it knows about the columns, it can handle niceties for you: for example, the first column is a string, the second is an integer, the third is dollar values. It can use that to make sure it returns those columns to you correctly formatted, and it can also store numbers more efficiently than just strings of digits.

In reality a modern database is doing a whole bunch of far more clever optimizations than just comma separated values but it's a mental model of what's going on that works fine. The data all lives on disk, often as one big file, and the DBMS caches parts of it in memory for speed. Sometimes it has different files for the data and the metadata, or for indexes that make it easier to find things quickly, but we can safely ignore those details.

RDBMS are older, so they date from a time when memory was really expensive, so they usually optimize for keeping most things on disk and only put some stuff in memory. But they don't have to: some RDBMS keep everything in memory and never write to disk. That makes them much faster!

Is it still a database if all the structured data stays in memory? Sure. It's a pile of structured data. Nothing in that definition says a disk needs to be involved.

So what does the "relational" part of RDBMS mean? RDBMS have multiple tables of data, and they can relate different tables to each other. For instance, imagine a new table called "Farmers":


and we modify the Fruits table:

Farmer IDFruitQuantityPrice

The Farmers table gives each farmer a name and an ID. The Fruits table now has a column that gives the Farmer ID, so you can see which farmer has which fruit at which price.

Why's that helpful? Two reasons: space and time. Space because it reduces data duplication. Remember, these were invented when disks were expensive and slow! Storing the data this way lets you only list "susan" once no matter how many fruits she has. If she had a hundred kinds of fruit you'd be saving quite a lot of storage by not repeating her name over and over. The time reason comes in if you want to change Susan's name. If you repeated her name hundreds of times you would have to do a write to disk for each one (and writes were very slow at the time this was all designed). That would take a long time, plus there's a chance you could miss one somewhere and suddenly Susan would have two names and things would be confusing.

Relational databases make it easy to do certain kinds of queries. For instance, it's very efficient to find out how many fruits there are in total: you just add up all the numbers in the Quantity column in Fruits, and you never need to look at Farmers at all. It's efficient and because the DBMS knows where the data is you can say "give me the sum of the quantity colum" pretty simply in SQL, something like SELECT SUM(Quantity) FROM Fruits. The DBMS will do all the work.

NoSQL databases

So now let's look at the NoSQL databases. These were a much more recent invention, and the economics of computer hardware had changed: memory was a lot cheaper, disk space was absurdly cheap, processors were a lot faster, and programmers were very expensive. The designers of newer databases could make different trade-offs than the designers of RDBMS.

The first difference of NoSQL databases is that they mostly don't store things on disk, or do so only once in a while as a backup. This can be dangerous – if you lose power you can lose all your data – but often a backup from a few minutes or seconds ago is fine and the speed of memory is worth it. A database like Redis writes everything to disk every 200ms or so, which is hardly any time at all, while doing all the real work in memory.

A lot of the perceived performance advantages of "noSQL" databases is just because they keep everything in memory and memory is very fast and disks, even modern solid-state drives, are agonizingly slow by comparison. It's nothing to do with whether the database is relational or not-relational, and nothing at all to do with SQL.

But the other thing NoSQL database designers did was they abandoned the "relational" part of databases. Instead of the model of tables, they tended to model data as objects with keys. A good mental model of this is just JSON:


Again, just as a modern RDBMS is not really writing CSV files to disk but is doing wildly optimized stuff, a NoSQL database is not storing everything as a single giant JSON array in memory or disk, but you can mentally model it that way and you won't go far wrong. If I want the record for Bob I ask for ID 0, Susan is ID 1, etc..

One advantage here is that I don't need to plan in advance what I put in each record, I can just throw anything in there. It can be just a name, or a name and an age, or a gigantic object. With a relational DB you have to plan out columns in advance, and changing them later can be tricky and time-consuming.

Another advantage is that if I want to know everything about a farmer, it's all going to be there in one record: their name, their fruits, the prices, everything. In a relational DB that would be more complicated, because you'd have to query the farmers and fruits tables at the same time, a process called "joining" the tables. The SQL "JOIN" keyword is one way to do this.

One disadvantage of storing records as objects like this, formally called an "object store", is that if I want to know how many fruits there are in total, that's easy in an RDBMS but harder here. To sum the quantity of fruits, I have to retrieve each record, find the key for fruits, find all the fruits, find the key for quantity, and add these to a variable. The DBMS for the object store may have an API to do this for me if I've been consistent and made all the objects I stored look the same. But I don't have to do that, so there's a chance the quantities are stored in different places in different objects, making it quite annoying to get right. You often have to write code to do it.

But sometimes that's okay! Sometimes your app doesn't need to relate things across multiple records, it just wants all the data about a single key as fast as possible. Relational databases are best for the former, object stores the best for the latter, but both types can answer both types of questions.

Some of the optimizations I mentioned both types of DBMS use are to allow them to answer the kinds of questions they're otherwise bad at. RDBMS have "object" columns these days that let you store object-type things without adding and removing columns. Object stores frequently have "indexes" that you can set up to be able to find all the keys in a particular place so you can sum up things like Quantity or search for a specific Fruit name fast.

So what's the difference between an "object store" and a "noSQL" database? The first is a formal name for anything that stores structured data as objects (not tables). The second is... well, basically a marketing term. Let's digress into some tech history!

The self-defeating triumph of MySQL

Back in 1995, when the web boomed out of nowhere and suddenly everybody needed a database, databases were mostly commercial software, and expensive. To the rescue came MySQL, invented 1995, and Postgres, invented 1996. They were free! This was a radical idea and everybody adopted them, partly because nobody had any money back then – the whole idea of making money from websites was new and un-tested, there was no such thing as a multi-million dollar seed round. It was free or nothing.

The primary difference between PostgreSQL and MySQL was that Postgres was very good and had lots of features but was very hard to install on Windows (then, as now, the overwhelmingly most common development platform for web devs). MySQL did almost nothing but came with a super-easy installer for Windows. The result was MySQL completely ate Postgres' lunch for years in terms of market share.

Lots of database folks will dispute my assertion that the Windows installer is why MySQL won, or that MySQL won at all. But MySQL absolutely won, and it was because of the installer. MySQL became so popular it became synonymous with "database". You started any new web app by installing MySQL. Web hosting plans came with a MySQL database for free by default, and often no other databases were even available on cheaper hosts, which further accelerated MySQL's rise: defaults are powerful.

The result was people using mySQL for every fucking thing, even for things it was really bad at. For instance, because web devs move fast and change things they had to add new columns to tables all the time, and as I mentioned RDBMS are bad at that. People used MySQL to store uploaded image files, gigantic blobs of binary data that have no place in a DBMS of any kind.

People also ran into a lot of problems with RDBMS and MySQL in particular being optimized for saving memory and storing everything on disk. It made huge databases really slow, and meanwhile memory had got a lot cheaper. Putting tons of data in memory had become practical.

The rise of in-memory databases

The first software to really make use of how cheap memory had become was Memcache, released in 2003. You could run your ordinary RDBMS queries and just throw the results of frequent queries into Memcache, which stored them in memory so they were way, WAY faster to retrieve the second time. It was a revolution in performance, and it was an easy optimization to throw into your existing, RDBMS-based application.

By 2009 somebody realized that if you're just throwing everything in a cache anyway, why even bother having an RDBMS in the first place? Enter MongoDB and Redis, both released in 2009. To contrast themselves with the dominant "MySQL" they called themselves "NoSQL".

What's the difference between an in-memory cache like Memcache and an in-memory database like Redis or MongoDB? The answer is: basically nothing. Redis and Memcache are fundamentally almost identical, Redis just has much better mechanisms for retrieving and accessing the data in memory. A cache is a kind of DB, Memcache is a DBMS, it's just not as easy to do complex things with it as Redis.

Part of the reason Mongo and Redis called themselves NoSQL is because, well, they didn't support SQL. Relational databases let you use SQL to ask questions about relations across tables. Object stores just look up objects by their key most of the time, so the expressiveness of SQL is overkill. You can just make an API call like get(1) to get the record you want.

But this is where marketing became a problem. The NoSQL stores (being in memory) were a lot faster than the relational DBMS (which still mostly used disk). So people got the idea that SQL was the problem, that SQL was why RDBMS were slow. The name "NoSQL" didn't help! It sounded like getting rid of SQL was the point, rather than a side effect. But what most people liked about the NoSQL databases was the performance, and that was just because memory is faster than disk!

Of course, some people genuinely do hate SQL, and not having to use SQL was attractive to them. But if you've built applications of reasonable complexity on both an RDBMS and an object store you'll know that complicated queries are complicated whether you're using SQL or not. I have a lot of love for SQL.

If putting everything in memory makes your database faster, why can't you build an RDBMS that stores everything in memory? You can, and they exist! VoltDB is one example. They're nice! Also, MySQL and Postgres have kind of caught up to the idea that machines have lots more RAM now, so you can configure them to keep things mostly in memory too, so their default performance is a lot better and their performance after being tuned by an expert can be phenomenal.

So anything that's not a relational database is technically a "NoSQL" database. Most NoSQL databases are object stores but that's really just kind of a historical accident.

How does my app talk to a database?

Now we understand how a database works: it's software, running on a machine, managing data for you. How does your app talk to the database over a network and get answers to queries? Are all databases just a single machine?

The answer is: every DBMS, whether relational or object store, is a piece of software that runs on machine(s) that hold the data. There's massive variation: some run on 1 machine, some on clusters of 5-10, some run across thousands of separate machines all at once.

The DBMS software does the management of the data, in memory or on disk, and it presents an API that can be accessed locally, and also more importantly over the network. Sometimes this is a web API like you're used to, literally making GET and POST calls over HTTP to the database. For other databases, especially the older ones, it's a custom protocol.

Either way, you run a piece of software in your app, usually called a Client. That client knows the protocol for talking to the database, whether it's HTTP or WhateverDBProtocol. You tell it where the database server is on the network, it sends queries over and gets responses. Sometimes the queries are literally strings of text, like "SELECT * FROM Fruits", sometimes they are JSON payloads describing records, and any number of other variations.

As a starting point, you can think of the client running on your machine talking over the network to a database running on another machine. Sometimes your app is on dozens of machines, and the database is a single IP address with thousands of machines pretending to be one machine. But it works pretty much the same either way.

The way you tell your client "where" the DB is is your connection credentials, often expressed as a string like "http://username:password@mydb.com:1234" or "mongodb://...". But this is just a convenient shorthand. All your client really needs to talk to a database is the DNS name (like mydb.com) or an IP address (like, plus a port (1234). This tells the network which machine to send the query to, and what "door" to knock on when it gets there.

A little about ports: machines listen on specific ports for things, so if you send something to port 80, the machine knows the query is for your web server, but if you send it to port 1234, it knows the query is for your database. Who picks 1234 (In the case of Postgres, it's literally 5432)? There's no rhyme or reason to it. The developers pick a number that's easy to remember between 1 and 65,535 (the highest port number available) and hope that no other popular piece of software is already using it.

Usually you'll also have a username and password to connect to the database, because otherwise anybody who found your machine could connect to your database and get all the data in it. Forgetting that this is true is a really common source of security breaches!

There are bad people on the internet who literally just try every single IP in the world and send data to the default port for common databases and try to connect without a username or password to see if they can. If it works, they take all the data and then ransom it off. Yikes! Always make sure your database has a password.

Of course, sometimes you don't talk to your database over a network. Sometimes your app and your database live on the same machine. This is common in desktop software but very rare in web apps. If you've ever heard of a "database driver", the "driver" is the equivalent of the "client", but for talking to a local database instead of over a network.

Replication and scaling

Remember I said some databases run on just 1 machine, and some run on thousands of machines? That's known as replication. If you have more than one copy of a piece of data, you have a "replica" of that data, hence the name.

Back in the old days hardware was expensive so it was unusual to have replicas of your data running at the same time. It was expensive. Instead you'd back up your data to tape or something, and if the database went down because the hardware wore out or something, then you'd buy new hardware and (hopefully) reinstall your DBMS and restore the data in a few hours.

Web apps radically changed people's demands of databases. Before web apps, most databases weren't being continuously queried by the public, just a few experts inside normal working hours, and they would wait patiently if the database broke. With a web app you can't have minutes of downtime, far less hours, so replication went from being a rare feature of expensive databases to pretty much table stakes for every database. The initial form of replication was a "hot spare".

If you ran a hot spare, you'd have your main DBMS machine, which handled all queries, and a replica DBMS machine that would copy every single change that happened on the primary to itself. Primary was called m****r and the replica s***e because the latter did whatever the former told it to do, and at the time nobody considered how horrifying that analogy was. These days we call those things "primary/secondary" or "primary/replica" or for more complicated arrangements things like "root/branch/leaf".

Sometimes, people would think having a hot spare meant they didn't need a backup. This is a huge mistake! Remember, the replica copies every change in the main database. So if you accidentally run a command that deletes all the data in your primary database, it will automatically delete all the data in the replica too. Replicas are not backups, as the bookmarking site Magnolia famously learned.

People soon realized having a whole replica machine sitting around doing nothing was a waste, so to be more efficient they changed where traffic went: all the writes would go to the primary, which would copy everything to the replicas, and all the reads would go to the replicas. This was great for scale!

Instead of having 1 machine worth of performance (and you could swap to the hot spare if it failed, and still have 1 machine of performance with no downtime) suddenly you had X machines of performance, where X could be dozens or even hundreds. Very helpful!

But primary/secondary replication of this kind has two drawbacks. First, if a write has arrived at the primary database but not yet replicated to all the secondary machines (which can take half a second if the machines are far apart or overloaded) then somebody reading from the replica can get an answer that's out of date. This is known as a "consistency" failure, and we'll talk about it more later.

The second flaw with primary/second replication is if the primary fails, suddenly you can no longer write to your database. To restore the ability to do writes, you have to take one of the replicas and "promote" it to primary, and change all the other replicas to point at this new primary box. It's time-consuming and notoriously error-prone.

So newer databases invented different ways of arranging the machines, formally called "network topology". If you think of the way machines connect to each other as a diagram, the topology is the shape of that diagram. Primary/secondary looks like a star. Root/branch/leaf looks like a tree. But you can have a ring structure, or a mesh structure, or lots of others. A mesh structure is a lot of fun and very popular, so let's talk about more about them.

Mesh replication databases

A diagram of a mesh topology I stole

In a mesh structure, every machine is talking to every other machine and they all have some portion of the data. You can send a write to any machine and it will either store it, or figure out what machine should store it and send it to that machine. Likewise, you can query any machine in the mesh, and it will give you the answer if it has the data, or forward your request to a machine that does. There's no "primary" machine to fail. Neat!

Because each machine can get away with storing only some of the data and not all of it, a mesh database can store much, much more data than a single machine could store. If 1 machine could store X data, then N machines could theoretically store N*X data. You can almost scale infinitely that way! It's very cool.

Of course, if each record only existed on one machine, then if that machine failed you'd lose those records. So usually in a mesh network more than one machine will have a copy of any individual record. That means you can lose machines without losing data or experiencing downtime; there are other copies lying around. In some mesh databases can also add a new machine to the mesh and the others will notice it and "rebalance" data, increasing the capacity of the database without any downtime. Super cool.

So a mesh topology is a lot more complicated but more resilient, and you can scale it without having to take the database down (usually). This is very nice, but can go horribly wrong if, for instance, there's a network error and suddenly half the machines can't see the other half of the machines in the mesh. This is called a "network partition" and it's a super common failure in large networks. Usually a partition will last only a couple of seconds but that's more than enough to fuck up a database. We'll talk about network partitions shortly.

One important question about a mesh DB is: how do you connect to it? Your client needs to know an IP address to connect to a database. Does it need to know the IP addresses of every machine in the mesh? And what happens when you add and remove machines from the mesh? Sounds messy.

Different Mesh DBs do it differently, but usually you get a load balancer, another machine that accepts all the incoming connections and works out which machine in the mesh should get the question and hands it off. Of course, this means the load balancer can fail, hosing your DB. So usually you'll do some kind of DNS/IP trickery where there are a handful of load balancers all responding on the same domain name or IP address.

The end result is your client magically just needs to know only one name or IP, and that IP always responds because the load balancer always sends you to a working machine.

CAP theory

This brings us neatly to a computer science term often used to talk about databases which is Consistency, Availability, and Partition tolerance, aka CAP or "CAP theory". The basic rule of CAP theory is: you can't have all 3 of Consistency, Availability and Partition Tolerance at the same time. Not because we're not smart enough to build a database that good, but because doing so violates physics.

Consistency means, formally: every query gets the correct, most up-to-date answer (or an error response saying you can't have it).

Availability means: every query gets an answer (but it's not guaranteed to be the correct one).

Partition Tolerance means: if the network craps out, the database will continue to work.

You can already see how these conflict! If you're 100% Available it means by definition you'll never give an error response, so sometimes the data will be out of date, i.e. not Consistent. If your database is Partition Tolerant, on the other hand, it keeps working even if machine A can't talk to machine B, and machine A might have a more recent write than B, so machine B will give stale (i.e. not Consistent) responses to keep working.

So let's think about how CAP theorem applies across the topologies we already talked about.

A single DB on a single machine is definitely Consistent (there's only one copy of the data) and Partition Tolerant (there's no network inside of it to crap out) but not Available because the machine itself can fail, e.g. the hardware could literally break or power could go out.

A primary DB with several replicas is Available (if one replica fails you can ask another) and Partition Tolerant (the replicas will respond even if they're not receiving writes from the primary) but not Consistent (because as mentioned earlier, the replicas might not have every primary write yet).

A mesh DB is extremely Available (all the nodes always answer) and Partition Tolerant (just try to knock it over! It's delightfully robust!) but can be extremely inconsistent because two different machines on the mesh could get a write to the same record at the same time and fight about which one is "correct".

This is the big disadvantage to mesh DBs, which otherwise are wonderful. Sometimes it's impossible to know which of two simultaneous writes is the "winner". There's no single authority, and Very Very Complicated Algorithms are deployed trying to prevent fights breaking out between machines in the mesh about this, with highly variable levels of success and gigantic levels of pain when they inevitably fail. You can't get all three of CAP and Consistency is what mesh networks lose.

In all databases, CAP isn't a set of switches where you are or aren't Consistent, Available, or Partition Tolerant. It's more like a set of sliders. Sliding up the Partition Tolerance generally slides down Consistency, sliding down Availability will give you more Consistency, etc etc.. Every DBMS picks some combination of CAP and picking the right database is often a matter of choosing what CAP combination is appropriate for your application.

Other topologies

Some other terms you frequently hear in the world of databases are "partitions" (which are different from the network partitions of CAP theorem) and "shards". These are both additional topologies available to somebody designing a database. Let's talk about shards first.

Imagine a primary with multiple replicas, but instead of each replica having all the data, each replica has a slice (or shard) of the data. You can slice the data lots of ways. If the database was people, you could have 26 shards, one with all names starting with A, one with all the names starting with B, etc..

Sharding can be helpful if the data is too big to all fit on one disk at a time. This is less of a problem than it used to be because virtual machines these days can effectively have infinity-sized hard drives.

The disadvantage of sharding is it's less Available: if you lose a shard, you lose everybody who starts with that letter! (Of course, your shards can also have replicas...) Plus your software needs to know where all the shards are and which one to ask a question. It's fiddly. Many of the problems of sharded databases are solved by using mesh topologies instead.

Partitions are another way of splitting up a database, but instead of splitting it across many machines, it splits the database across many files in a single machine. This is an old pattern that was useful when you had really powerful hardware and really slow disks, because you could install multiple disks into a single machine and put different partitions on each one, speeding up your achingly slow, disk-based database. These days there's not a lot of reason to use partitions of this kind.


That concludes this impromptu Databases 101 seminar! I hope you enjoyed learning a little bit more about this fantastically fun and critically important genre of software.

Biographies of every US president as audiobooks

posted 27 September 2019, updated 31 October 2019

I enjoy reading history, especially in the form of historical biographies. Over the years I've picked up books about several of the founding fathers of the USA, plus major historical figures like FDR, and eventually I realized that I'd covered a big chunk of all the presidents of the USA. So then I went back and decided to fill in all the gaps. I stopped at George HW Bush at the entirely arbitrary line of "latest dead president".

As I've been reading I started tweeting about fun facts about the presidents as I encountered them. Slowly that evolved into longer and longer threads about each president, and some of those threads were quite popular.

So here for ease of consumption is the full list of presidents I've read, which biography I read, a single sentence on what I thought about that biography and if I recommend it, as well as (where available) a link to the tweet or thread about that president.

If this list format doesn't grab you, it's also available as a spreadsheet.

  1. George Washington

    Washington: A Life by Ron Chernow

    Recommended. Fantastic. Deeply researched, engaging, readable, well paced.

  2. John Adams

    John Adams by David McCullough

    Recommended. Does its very best with a dull as dishwater subject. Detailed and well-written but a slog.

  3. Thomas Jefferson

    Thomas Jefferson: The Art of Power by Jon Meacham

    Recommended. Brilliantly written, full of great detail, pulls its punches on some of his darker aspects but doesn't ignore them.

  4. James Madison

    The Three Lives of James Madison: Genius, Partisan, President by Noah Feldman

    Recommended. Detailed and engaging, a little longer than its subject deserves.

  5. James Monroe

    The Last Founding Father: James Monroe and a Nation's Call to Greatness by Harlow Giles Unger

    Recommended. Concise, about as long as this shitty dude deserved, but too nice to him.

  6. John Quincy Adams

    John Quincy Adams: Militant Spirit by James Traub

    Recommended. JQA was even duller than his dad. The bio is workmanlike with much less of the color of other bios.

  7. Andrew Jackson

    Andrew Jackson: His Life and Times by H W Brands

    Recommended. Military-focused, containing less personal detail than is probably warranted, but very readable.

  8. Martin Van Buren

    Martin Van Buren: A Captivating Guide to the Man Who Served as the Eighth President of the United States by Captivating History

    Not recommended. Short and cliff notes-y but with some fun details, and very engaging. Best of the cliff-notes bios.

  9. William Henry Harrison

    William Henry Harrison: The Life and Legacy of the First American President to Die in Office by Charles River Editors

    Not recommended. Just under 2 hours long, a cliff-notes bio. Not great.

  10. John Tyler

    John Tyler by Robert J Spitzer

    Not recommended. 28 minutes long, a shitty all-the-presidents entry.

  11. James K Polk

    Polk: The Man Who Transformed the Presidency and America by Walter R Borneman

    Recommended. Author wanted to write about Jackson and sort of accidentally wrote about Polk. Not good.

  12. Zachary Taylor

    Zachary Taylor by Paul Finkelman

    Not recommended. A 24-minute all-the-presidents entry. Taylor sucked but so does this bio.

  13. Millard Fillmore

    Millard Fillmore by Joseph F Rishel

    Not recommended. 17 minutes long! The shortest bio yet, a worthless all-the-presidents entry.

  14. Franklin Pierce

    Franklin Pierce by Paul Finkelman

    Not recommended. Another all-the-presidents entry. A few fun tidbits.

  15. James Buchanan

    Worst. President. Ever. James Buchanan, the POTUS Rating Game, and the Legacy of the Least of the Lesser Presidents by Robert Strauss

    Recommended. A delightful treatment of a shitty subject, light-hearted but well researched, very engaging.

  16. Abraham Lincoln

    Team of Rivals: The Political Genius of Abraham Lincoln by Doris Kearns Goodwin

    Recommended. DKG has a well-earned reputation as an excellent biographer and Lincoln is an amazing subject. 42 hours of material fly by.

  17. Andrew Johnson

    After Lincoln: How the North Won the Civil War and Lost the Peace by A J Langguth

    Not recommended. Not really a bio of Johnson, but covers the aftermath of Lincoln so mostly about Johnson. Good but little personal detail about Johnson for that reason.

  18. Ulysses S Grant

    American Ulysses: A Life of Ulysses S. Grant by Ronald C White

    Recommended. Fantastic book of a truly great and under-rated president. Great details.

  19. Rutherford B Hayes

    Rutherford B Hayes by Hans L Trefousse

    Not recommended. 5 hours long which is 4 hours more than its subject deserved. Yawnfest.

  20. James A Garfield

    James A Garfield: The 20th President's 200 Days in Office by in60Learning

    Recommended. An hour long but left me wanting much more.

  21. Chester A Arthur

    The Unexpected President: The Life and Times of Chester A. Arthur by Scott S Greenberger

    Recommended. Recommended only for the excellent details of Julia Sand's letters.

  22. Grover Cleveland

    An Honest President: The Life and Presidencies of Grover Cleveland by H Paul Jeffers

    Not recommended. A mediocre bio of a mediocre guy.

  23. Benjamin Harrison

    Benjamin Harrison by Allan B Spetter

    Not recommended. An all-the-presidents series entry, just 22 minutes long, barely worth it.

  24. [Cleveland again]

  25. William McKinley

    William McKinley by Kevin Phillips, Arthur M Schlesinger

    Not recommended. An all-the-presidents entry much longer than it needed to be.

  26. Theodore Roosevelt

    The Rise of Theodore Roosevelt by Edmund Morris

    Recommended. The first of an epic 3-book series on TR of which I read 2. Definitive and captivating.

  27. William Howard Taft

    William Howard Taft: The American Presidents Series: The 27th President, 1909-1913 by Jeffrey Rosen, Arthur M. Schlesinger Jr., Sean Wilentz

    Not recommended. Lacks compelling personal details that could have fit in 5 hours. Not compelling.

  28. Woodrow Wilson

    Woodrow Wilson: A Biography by John Milton Cooper

    Recommended. Way too long given how dull Wilson is but well-written.

  29. Warren G Harding

    Warren G Harding by James D Robenalt

    Not recommended. Short and dull but still not as bad as The Bloviator, which is awful and nobody should read.

  30. Calvin Coolidge

    Coolidge by Amity Shlaes

    Recommended. A concise and insightful look at an important transitional president.

  31. Herbert Hoover

    Hoover: An Extraordinary Life in Extraordinary Times by Kenneth Whyte

    Recommended. I could have read another 50% of this 27-hour book. Fascinating.

  32. Franklin D Roosevelt

    FDR by Jean Edward Smith

    Recommended. A gem of a book, full of amazing details, and a wonderful subject.

  33. Harry S Truman

    Harry S Truman: A Life by Robert H Ferrell

    Recommended. Well-written and detailed but insufficiently critical of its surprisingly crooked subject.

  34. Dwight D Eisenhower

    Eisenhower in War and Peace by Jean Edward Smith

    Recommended. Sort of a side-effect of the FDR bio but well-written and workable.

  35. John F Kennedy

    An Unfinished Life: John F Kennedy, 1917-1963 by Robert Dallek

    Recommended. A good book but cruelly, pointlessly abridged, please release the full thing.

  36. Lyndon B Johnson

    Lyndon Johnson and the American Dream: The Most Revealing Portrait of a President and Presidential Power Ever Written by Doris Kearns Goodwin

    Recommended. A mind-blowing book of amazing depth and quality of writing from an author incredibly close to her subject. Unmissable.

  37. Richard Nixon

    Richard Nixon: The Life by John A Farrell

    Recommended. Detailed and pulls few punches, full of great personal details.

  38. Gerald Ford

    Ambition, Pragmatism, and Party: A Political Biography of Gerald R. Ford by Scott Kaufman

    Recommended. Ford truly sucked, and this comprehensive but mercifully concise book covers it well.

  39. Jimmy Carter

    President Carter: The White House Years by Stuart E. Eizenstat

    Recommended. This bio is WAY TOO LONG for the amount of interest; a book half the length would be fine.

  40. Ronald Reagan

    Reagan: The Life by H W Brands

    Recommended. Solid and comprehensive; a better book will be written when more stuff is declassified.

  41. George H W Bush

    Destiny and Power: The American Odyssey of George Herbert Walker Bush by Jon Meacham

    Recommended. Comprehensive, hagiographic, far too long.

I will not be doing Bill Clinton, George W Bush, Barack Obama or Trump, as they are all still too recent for biographies to have all the facts.

Becoming American

posted 07 September 2018
A small american flag and a pink button saying 'freedom protector'

Today I became an American citizen.

The ceremony itself is, well, very American. Me and 854 other soon-to-be citizens assembled at the Paramount Theatre in Oakland, clutching our precious green cards and the flimsy single-page letter we received in the mail saying we had passed all the tests needed to become citizens. As we filed in, a choir of elderly volunteers on stage sang various patriotic songs, with more enthusiasm than talent.

At the door, you hand them your green card -- a terrifying event; my green card is a precious document that cost me 5 years and well upwards of $25,000 in fees to acquire -- and in exchange you get a tiny American flag on a stick, and an envelope. In the hallway, a massive team of USCIS officials work in parallel to process all 855 applications at once, so that by the end of the ceremony they can hand you a certificate of naturalization.

A USCIS official who had obviously done this many, many times -- there are two ceremonies per month, and he looked like he'd worked there a long time -- ran through the program. We'd hear how to register to vote, how to update social security, how to apply for a passport. Then we'd be led through the Oath of Allegiance, which is the actual point at which you become legally a citizen. He encouraged people to feel free to clap and cheer, and the audience responded enthusiastically, frequently accompanied by waving all the little flags we'd been given.

He had a bunch of little jokes, obviously time-worn. He talked about all the countries participating, and rattled off welcomes in Spanish, then French, then -- prompting gasps of increasingly impressed surprise -- Chinese, Hindi and Taglog. These corresponded to the biggest countries of origin -- China was the biggest, then Mexico, Nicaragua, India, the Philippines and Canada. He listed all the countries participating in alphabetical order and got people to stand up when their country was called.

He thanked us for coming to America and strongly encouraged us to register to vote. He skirted as close as I can imagine he was professionally able to pointing out that America has a lot of problems right now, and a bunch of new voters might go some way to fixing that.

Then somebody came on and sung the national anthem -- we were encouraged, but not required, to sing along, and the crowd enthusiastically joined in -- and then we administered the oath, which took all of 45 seconds. Then a second, even shorter oath for those who intended to apply for a passport that day. Then a video message from Madeleine Albright, talking about how proud she was to have risen from refugee to Secretary of State, a second video from Donald Trump, who unconvincingly espoused the virtues of immigration, and then another very patriotic video of multi-cultural people waving flags.

Then they played "Proud to be an American", which is an aggressively condescending and arrogant song, but yet again most of the crowd sang along enthusiastically. Then the officials who'd been frantically printing certificates in the hall filed in and very efficiently handed them out. There was a lot of cheering, more flag waving, tons of selfies. And that, about 2 hours after we'd started, was that.

The ambivalent American

Growing up in my family, America was not the shining land of the free. Americans, according to my family, were definitely The Worst. They were loud, boorish, arrogant, rude, uncultured. My family, who watch every televised sporting event of any kind up to and including sheep herding trials, would not watch American football, baseball, or basketball. Liking American TV and movies was considered letting people down. As a child, even liking Mickey Mouse was considered shamefully unpatriotic.

But of course we did watch American movies and TV, because that was most of what was available. We consumed American culture while vilifying Americans all the while. There was no shortage of hypocrisy in this.

But the rest of the world and especially the Caribbean has a lot of very justifiable reasons to be unenthusiastic about American hegemony. American drug policy is responsible for political and economic disaster across all the countries south of the border. American culture is, in fact, violent, materialistic, and full of unhealthy and contradictory messages about bodies, food, beauty, religion and more.

So when my career took me to America my family were genuinely aghast. You want to move to *America*? But there are Americans there! It'll be awful! How can you stand being surrounded by Americans all the time? I tried to explain that not all Americans are like the terrible Americans who take cheap vacations to the Caribbean, and that there are many parts of America that are beautiful and cultured. My family, whose primary experience of America is visiting some relatives of ours who live in south Florida, were unconvinced. America, as far as they were concerned, was an un-ending series of strip malls, shitty chain restaurant food, and rednecks.

I arrived in the USA in 2007, and worked hard as a volunteer to elect Barack Obama in 2008. Here was a vision of America I could be proud of: diverse, caring, cultured, humble and respectful of the rest of the world. Obama's presidency had a great number of flaws, but I was a fan all the way to the end.

Having originally planned to stay only a few years, I hung around. I swapped between a few visas and, after five years and a huge amount in lawyers fees mostly paid by my employers, acquired a green card. You have to wait 5 years after getting a green card to become a citizen, so I had a few years to decide if, in far-off 2017, I wanted to become a US citizen.

Then the 2016 election happened.

A flight to safety

A week after becoming president, Trump signed Executive Order 13769, usually called the Muslim Ban. Quite apart from the horrifically cruel and transparently bigoted nature of the order, it included a side-effect that was particularly horrifying to green card holders: the ban apparently applied to us too. Green-card holders were initially denied entry. Some were coerced into signing documents that forced them to relinquish their precious green cards.

After fighting so hard and so long to get our green cards, the idea that they could be stripped away was unprecedented, shocking, and filled me and my fellows with fear. There was a mad rush to apply for citizenship. Green card holders who'd been sitting on the cards for years not applying for citizenship suddenly were desperate for the additional security it afforded. Unable to apply until August 2017, I watched helplessly as the queue for citizenship lengthened from 3 months to a year. Equally fearful, I applied as soon as I could.

So now I find myself a citizen. It's a hard time to feel excited about that. The country is sliding into fascism (though in fairness so apparently is much of Europe). Inequality is rising. Racial injustice is omnipresent -- though, again in fairness, it was always there and has just recently become visible. There is much that is broken about America.

It would be easy to rationalize my citizenship to myself and my now even more horrified family as a mere administrative convenience, a security device to keep this dangerously capricious administration from summarily deporting me. But easy as that would be, it would be false.

More American than I thought

There is, as I kept telling my family, a lot to like about America. There's natural beauty, friendly people, culture. There is a sense of possibility, an openness to trying new things that I never found in the seven years I lived in the UK. There's institutions and a respect for the rule of law that Trinidad, though it will always be home to me, increasingly lacks.

There's potential to America, potential it's not currently living up to, but that remains tantalizingly close. A new generation are rising who genuinely value the diversity that makes America better, who understand that there is virtue and strength in taking from the richest to help the poorest, who realize systems like healthcare and gun laws that make America a horrifying outlier amongst rich nations can be changed for the better.

The day after the election, I was walking through downtown San Francisco and passed through one of the many spontaneous protests that were happening that week. A woman in the crowd handed me the small pink button in the picture above. It says "freedom protector". You're supposed to wear it to indicate that you will fight to keep others safe. I knew I couldn't wear it, because I couldn't fight. Even being near a protest was a risky thing for a green card holder.

But I held onto it. I kept it in the pocket of my favorite hoodie like a talisman, for the 21 long months between the nightmare of election night and today being finally safe from the whims of a dangerously unhinged executive. I was not a freedom protector, but that was the day that I decided I wanted to be. Tomorrow I get to wear the badge. Finally safe, tomorrow I can start protecting others.

I haven't become American just to stay safe. I've become American because after 11 years, Americanness has seeped into me. I'm choosing to be here because I want to help. I'm choosing to be here because I am, no matter how bad things are right now, fundamentally optimistic about the future of the country. I think America can be better, and I think I can do something about that. And that's a very American way to think.


posted 12 June 2018

Imagine you lived in a world where all there was only one kind of shoe, in a single size, called "medium". Everybody wears these shoes and no other shoes are available.

Most people do in fact have medium-sized feet. For most people this world would be totally fine, and since there was only one kind of shoe available, they'd probably not think about their shoes very much. They've got some shoes, no further decisions to be made.

But it wouldn't work for everybody.

Some people have smaller feet. People with smaller feet would find the shoes really annoying. For people with slightly smaller feet, the shoes might just flop around uncomfortably. For people with really small feet the shoes would fall off all the time. They'd get annoyed with the stupid shoes and they'd stuff newspaper into the toes to make them fit. It would work, and probably nobody would even be aware that their shoes don't fit properly.

Some people have bigger feet. People with only slightly larger feet would find the shoes uncomfortable. They might get calluses, or the heel might cut up their ankle and hurt a lot, but not so much they couldn't walk. But people with really big feet would find the shoes excruciating. Every step would be agony. They might want to stop walking altogether.

This is how I've come to understand gender. Most people, the gender they were handed at birth suits them just fine, and they don't even think about it. For other people, their gender isn't great but they can make it work with some extra effort. But for others, their gender is excruciating. Every step reminds them that these shoes don't fit, that something is wrong, that they need to change things.

The revolution in gender identity that's going on isn't just about the people who identify as trans, the people whose shoes fit so badly they had to change them no matter the cost. It's the realization for everybody that shoes come in every size. Maybe your shoes are fine, but it gives you the opportunity to explore: what if a couple sizes larger or smaller? Would that be more comfortable? What if I don't want to wear any shoes at all?

And of course, not only can we get shoes in different sizes, we can get them in different colors and styles. Some shoes look kind of ridiculous, especially if you've only ever seen standard, medium-sized shoes up until now. But for some people some brightly-colored new shoes are just what they've always needed. And why should anyone object? If somebody else changes their shoes it doesn't do anything to the shoes you're wearing, except maybe make you wonder if you need some new ones.

My shoes didn't always fit perfectly, so I got a slightly different size twenty years ago. Nobody noticed the shoes had changed, but they did notice that I walked more easily.

How are your shoes?

tagged with

Life in the bonus round

posted 26 May 2018

I've written about my teenage years several times before: about how my childhood shaped my choice of career and how the internet helped me connect with people and how it got better. I've mentioned in passing that I was, as a teenager, depressed and suicidal. But I've never written about how that period changed me in a very specific and important way: I live my life in a bonus round. It's fundamental to the way I think about myself, so I wanted to write it down.

Firstly: there is a different between being suicidal and just being a moody teenager. I was the first kind. I had written the notes (lots of them), and I had not just a plan but a backup plan. I also had a specific date: the day before my sixteenth birthday.

In the end I didn't do it. That day wasn't too bad, so I didn't think about it that day, and then I'd missed the date. But the intention to do it was so strong, so clear, I had spent so much time imagining exactly how it would go -- the rush of the air, the crunch as my neck hit the concrete -- that it made me think: what if I *had* done it?

What if I did die that day? I don't believe in any kind of afterlife, so death is just the end. Everything that's happening to me, good or bad, ends. All the dials go to zero. And I thought: what if I could then, at that point, make a decision about whether to keep going? Like in a video game, where you die or the game ends but it gives you the option to keep playing. What if life had a bonus round?

In a bonus round it doesn't matter how you do. There's no score, you can't win or lose, you're just playing for the fun of playing. You can stop any time. There's no time pressure and no ranking.

See, until that point in my life I had been judging myself relative to a pretty strictly defined idea of my own future. I was a straight white male in good health, from a well-off family and excellent opportunities for education. I knew how my life was going to go, and it was going to be great.

Discovering I was gay changed that. I still had a lot of internalized homophobia back then, so my idea of what being gay meant was a lot harder than my life has actually turned out to be, but there's no denying that being gay threw this perfect life I'd imagined for myself off the tracks it had been running on. There were roadblocks and obstacles I'd never expected. The game was no longer on the easiest setting. Don't get me wrong, my life has still been really easy relative to most people, but it was hard to think about it like that back then, when I was weeping myself to sleep every night, mourning my future.

But thinking about it as a bonus round allowed me to change my perspective. Okay, life #1, the perfect one, that was gone. That kid killed himself. Now it's bonus round. Everything that happens now just happens for fun. Every achievement can be measured relative to "nothing happened, because you were dead". Doing anything at all is better than doing nothing, no matter how well or how badly it goes. And if it ever becomes unbearable, well, you already died. That option remains available.

Does this sound sad or scary to you? I don't really know how it sounds to somebody else. But to me it's always been a comfort. In this life the scoreboard is turned off. I can't win or lose. I'm just playing for the fun of it. My life is, forever, the bonus round, and I'm still enjoying the game.

tagged with

Hands to Heaven, but about web standards

posted 11 April 2018

A million years ago I happened across a blog post called A Love Song About Web Standards. It featured a song called "Hands to Boag", a perfectly reproduced cover of the 80s song "Hands to Heaven", but rewritten to be about web standards and web development and the blog Boag, which is about those things. Even if you have never heard of Boag it's a pretty great cover and should make most web developers smile.

It's pretty good, but even better when you learn that Marcus Lillington, a host of the podcast in question, used to be a member of the band Breathe, who originally sang Hands to Heaven.

Every so often this idea that an 80s superstar is a web developer and people remade his song to be about web development returns to my brain and I go back and find the song and laugh again. But the last time, I discovered bit-rot had set in and the MP3 was no longer playable on their site. This is a tragedy! So I have rescued it and put it here, so you can still enjoy it:

Quasar Industries' Domestic Android robot hoax

posted 26 March 2018
In the late 1970s a shady company called Quasar Industries, capitalizing on popular enthusiasm for robots following the release of Star Wars, rolled out an elaborate hoax. It was a robot that purported to have speech recognition and be capable of dozens of household tasks, including vacuuming the living room and walking the dog. In fact it was all just marketing and puppetry. More interesting is that it spurred a debate in the barely-formed Internet community at the time about free speech. This excerpt from the excellent book Where Wizards Stay Up Late, about the birth of the Internet, is fascinating:

Then, in the spring of 1977, Quasar rolled in the door. Its arrival marked the beginning of the first debate over free speech in cyberspace. The controversy centered on an unusual device made by Quasar Industries and blew up into an argument over using the taxpayer- fundedARPANETto speak, in openly critical terms, about a private company.

The brainchild of Quasar Industries, the device stood five feet four inches and weighed two hundred forty pounds. It was called the Domestic Android robot, a programmable helper that could perform a dozen basic household tasks such as mopping the floor, mowing the lawn, washing dishes, and serving cocktails. It came equipped with a personality and speech, so that it could “interact in any human situation.” It could “teach the kids French” and “continue teaching them, while they sleep.” At the advertised price of $4,000, the thing seemed a steal.

Phil Karlton of Carnegie-Mellon was the first to alert the Msg-Group, on May 26, 1977. His site on theARPANETwas heavily involved in exploring artificial intelligence, speech recognition, and related research problems, so he knew a thing or two about robots. The android and its inventor had attracted a fair amount of national press attention, most of it favorable. Quasar’s sales pitch had also caught the attention ofConsumer Reports,which ran a skeptical item on it in the June issue, just out.

At first Quasar seemed nothing but an amusing diversion from the MsgGroup’s main business. Everyone in the group knew the thing was a hoax, and for a while that seemed enough. But then a sense of civic duty arose. Dave Farber told of being in Boca Raton, Florida, and hearing on the radio that the Dade County police department was considering purchasing a Quasar guard robot for their county jail, for $7,000. In March theBoston Globeran a story quoting MIT’s Marvin Minsky and other skeptical AI experts. But the article took the overall attitude, said a MsgGroup member, that it “just goes to show you, those academicians can’t do anything practical, and all you need is some guy working in the back of a garage to put them to shame.” The saga left a trail of disbelief in the artificial intelligence research community.

Brian Reid and a colleague, Mark Fox, from the Carnegie-Mellon Artificial Intelligence Lab, posted an offbeat report to everyone in the MsgGroup, giving them a personal account of their inspection of the domestic robot, “Sam Strugglegear,” at a large department store in downtown Pittsburgh. People in the research community, knowing of CMU’s pioneering AI work, had been calling the Lab to ask how it was possible for Quasar’s robot to be so much better at speech recognition than anything CMU had produced. Rising to the challenge, a four-member team from CMU had done the fieldwork.

“They found a frightening sight,” reported Reid and Fox. In the men’s department, among the three-piece suits, was a five-feet-two-inch “aerosol can on wheels, talking animatedly” to a crowd. Electric motors and a system of gears moved the device’s arms. The robot seemed conversant on any subject, recognized the physical features of customers, and moved freely in any direction. The crowd was charmed.

But the scientists were skeptical. They looked around for some evidence of a remote controller. “Lo and behold, about ten feet from the robot, standing in the crowd, we found a man in a blue suit with his hand held contemplatively to his mouth like Aristotle contemplating the bust of Homer in the famous Rembrandt painting.” Reid and the others watched for awhile and noticed that whenever the robot was talking, so was the man in the blue suit—muttering into his hand. The man had a wire dangling suspiciously from his waist.

The discussion about the Quasar robot continued on and off for a couple of years until in early 1979, Einar Stefferud, the MsgGroup’s moderator, and Dave Farber, who had been lurking on the sidelines of the commentary, sent a note of caution to the MsgGroup. “We are asking for potential problems,” they warned, “when we criticize the Quasar robot.” Using U.S. Government facilities to cast aspersions on a corporation, they said, could backfire on the ARPA research community. They urged their peers to impose careful self-censorship, to report only facts of technical interest to the community. Not everyone agreed, and with that the MsgGroup got embroiled in a soul-searching exchange.

Help decriminalize homosexuality in Trinidad and Tobago

posted 12 December 2017

The country where I was born and grew up still has some terrible, outdated laws against homosexuality. The laws are seldom enforced, but hang over the heads of gay people in T&T, forcing them to be quiet, subjugated, and fearful. They live their lives as second class citizens before the law. It is the reason I can't live in the country I was born in. These laws create misery for tens of thousands of people.

I have been waiting a long time for somebody to step up to the challenge of taking on these cruel and repressive laws, and finally, that's happening:

Jason Jones, a Trini based in London and an activist for 28 years, filed a legal challenge to these laws in February 2017, and the case will be heard in January of 2018.

The government is defending the law. Jason has received dozens of death threats and has had to hire bodyguards to protect him. But he's sticking to his guns, and his bravery for a cause that is so close to my heart is inspiring.

Jason isn't getting anything out of this case. There's no payout for him if he wins, just the freedom for him and the thousands of other gay Trinidadians to be openly themselves in the country they were born in. His legal team is working for free. Nevertheless, there are legal fees, travel and security expenses to cover. He's raising funds, and I want you to donate to his campaign right now. Over on Twitter, I'm going to be encouraging people to donate by matching funds.

Are we making the web too complicated?

posted 21 May 2017

Everyone's favorite sarcastic talking pushpin asked an honest question about the state of the web:

I replied with a tweetstorm. Here it is as a slightly more readable blog post on my ancient, creaky blog.

The replies to Maciej's tweet are interesting to read. They fall roughly into two camps:

  1. Older/not front-end developers: because the web is shit!
  2. Current front-end developers: because shit is hard!

As is often the case, both camps are correct! The web is a shitshow of wheel reinvention and bad APIs. It's also a blizzard of innovation.

Expectations for what a web site should be able to do have evolved enormously. Users expect snappy, desktop-like responsiveness and rich presentation in web apps. They also expect those same web apps to work equally well on mobile devices. And they expect these apps to load basically instantly. As Tom Dale says, that's actually a harder problem than desktop or mobile apps face. Users expect to download and install those types of apps before they will run.

Devs must meet these high expectations, but they have no more time and no more co-workers than they did before, and they still have to ship just as fast. To meet this requirement, they are throwing ever larger combinations of frameworks, boilerplate code, tools and build chains at the problem. The result is a lot of complexity and it's frequently frustrating.

The web as a platform, as ever, lags behind its developers. They want ES6 syntax, they want modular JavaScript, modular CSS and modular HTML. Browsers provide none of these things, so they are hacked together in frameworks.

Devs aren't adopting all this new tech just because it's new and fun (though it's a bad idea to dismiss fun as a valuable quality in a development tool). They are doing it because they have to in order to survive.

Modern web dev has the complexity of, say, the native mobile ecosystem, but no single vendor or consensus build chain or tools. iOS and Android have compilers and long build steps and dozens of competing frameworks, but nobody complains that these things are unnecessary. But they do for the web, because the web didn't need them before. (And if you're okay with throwing together a simple, 2000s-era web page, they're still not. But few users are happy with those any more.)

Are there some people using a huge pile of JavaScript and a monstrous build chain to throw together a single-pager web site with one box that collects an email address? For sure. And that's silly and unnecessary. But so what? The misuse of technology does not invalidate it.

Over the next 5 years I expect there will be a lot of consolidation in technologies and in tools. A lot of the stuff web devs are currently constructing using frameworks will be adopted as first-class citizens, built into browsers. Because the web has a lot of inertia, people will keep using these tool chains longer than they need to, much as web devs still use jQuery even though mosts of its API is part of browsers natively now.

jQuery is an instructive example: jQuery was a reaction to a terrible API and a lack of raw functionality in the web at the time. It changed the web for the better, forever, by showing browser makers what devs needed, and how it should work. It was a huge success, and its ultimate success is that it made itself unnecessary.

Webpack, babel, react, and the cambrian explosion going on in that ecosystem will do the same thing again. All of these frameworks and tools are devs experimenting, seeing where they want the web to go.

jQuery was a performance problem at scale, and sometimes over- or mis-applied, or used superfluously by newbies who didn't know there was a simpler way. Big deal.

Modern front-end devs are repeating the same mistakes: these new frameworks often create shocking performance hits. Sometimes we over-use them. Sometime we mis-use them. But the web is evolving, and evolution is by nature slow and messy. And the results are already amazing.

Is modern web development fearsomely, intimidatingly complicated? Yes, and that's a problem. Will we make it simpler? Definitely, but probably not as soon as you'd like. Is all this new complexity worthwhile? Absolutely.

The web's amazing capacity to reinvent itself, to evolve and adapt to new needs is its strength as a platform. Things that don't adapt die or are replaced. Nobody is talking about the death of the web. Nobody is demanding it be replaced. If anything they're pleading for it to slow down a bit, and let them catch up. That's a sign of a healthy platform, an innovative platform. The web was like that in 1996 and it's still like that, and that's amazing.

Nobody but nobody loves the web more than I do. It's my baby. And like a child, it's frustrating to watch it struggle and make mistakes. But it's amazing to watch it grow up.

How does one defeat Donald Trump?

posted 06 February 2017

It's a question on the minds of all right-thinking Americans, and on mine. I don't claim to be a political genius, or that this is the right solution or the only solution. But here's what I've got so far; tell me what you think.

First, to defeat Donald Trump you must make him unpopular. His popularity is what elected him, but more importantly it is what drives him. An unpopular Donald Trump will melt down and quit, humiliated. That's what we want. We need "being the next Donald Trump" to be an ignominious fate. We want Donald to never run again, and we want any Donald-shaped monster in future to be terrified of the possibility.

To work out how to make him unpopular, we must understand what made him popular in the first place. To do this, we must look beyond our liberal peers, with whom he is already maximally unpopular, as widespread demonstrations have indicated.

Here's what Donald Trump supporters believe about him (however incorrectly) that they like:

  • he is not part of the establishment
  • he is not as corrupt as most politicians
  • he is secretly racist, just like them, and will look out for white people
  • he's a strong man who will keep them safe

Anything about economic insecurity and not listening to the concerns of rural voters or the working class is bullshit. Poor people did not vote for Trump (his voters earn above the median wage, unlike Hillary). There are also plenty of rural voters who went for Hillary, and plenty of city dwellers who voted for Trump.

So to defeat Trump, we must make him look highly corrupt, part of the establishment, unwilling or unable to privilege white people over other people, and weak. The first two are easy, the second two more challenging.

Part of the establishment

It's easy to make Trump look like part of the establishment because he is part of the establishment. He is the damn president. Everything bad that the government does, whether or not he had anything to do with it, can and should be tied to him. Every slip in the economy, every problem with healthcare, every sparrow that falls from a tree should be loudly attributed to Donald's mismanagement. Republicans got really good at this and so should we. We can also point to his appointments of CEOs of Exxon, Goldman Sachs and other very-much-establishment companies to important posts.


Making him look more corrupt than most politicians is trivial because he is shockingly corrupt. His casino deals, his mafia ties, his many bankruptcies, Trump University, and repeatedly welching on debts to contractors establish him easily as a cheat, a liar and a crook. His base seemed to overlook or ignore these things as "tough negotiation" or something but we can keep dredging up more tales of his thievery basically forever.

Not racist?

Making him look like he's not secretly racist is very tricky because he is openly racist. He has repeatedly said he believes his superior genes guarantee his success, and that's before you get to his many obviously racist acts, from refusing to rent houses to black people, to failing to condemn the KKK, to demonizing Mexicans and Muslims (neither of which is a race, but racists aren't very bright).

To make Trump look un-racist will not work. What we can do is make him look powerless to act on that. Here we've already seen the most action: protests on the Muslim ban and swift legal action have halted it and may overturn it entirely. Protests against his border wall will likewise do so, and legal or legislative action should come there too. Making him look powerless to enact racist policy, however, is just part of the bigger play: make him look weak.


Because by far what people responded to in Trump is his strong man persona. He claimed he could do anything, fix anything, build anything, and it would all be great, the best, yuge, people would love it. His supporters bought these empty boasts as promises, so we have to puncture the idea that he can get anything done.

We have to be careful though. If the obstruction appears to come from outside -- from filibusters and other legislative hacks, from well-meaning heroics by democratic appointees, from "the establishment" -- then we bolster his support rather than erode it. Instead it needs to come from within, and here we are aided by Trump's stupidity and incompetence. His failure to negotiate trade deals, his inability to fund the wall, his botched attempt to ban Muslims, his failure to deport illegal immigrants: these are or will soon be his failures, and we can amplify them. This house of cards will collapse on its own, but we need to make sure it falls our way.

The other way to make him not just look weak but really become weaker is to peel off his inner circle one by one. The loathsome, openly racist and anti-semitic Bannon has already overstepped several times, to Trump's displeasure. By amplifying Trump's sense that he is being manipulated and overshadowed, we can use Trump's own ego to get Bannon ejected or diminished.

The repugnant Kellyanne Conway, with her "alternative facts" and imaginary terrorist attacks, is also faring badly in the spotlight. In any other administration her repeated, obvious lies would have already had her fired. In the Trump administration what will get her fired is if news organizations refuse to interview her anymore because her every word is openly mocked. If she doesn't get to speak on television, her power and her value to the administration will fail and she will be discarded after one lie too many. Sean Spicer will suffer a similar fate, perhaps even sooner.

Next steps

What can we do? We can amplify. Every failure must be trumpeted, every policy overturned, every decision nullified by protest or local action. His inner circle must be hounded until they become political liabilities, leaving Trump isolated and impotent. But be selective: don't amplify things that make liberals angry (you'll exhaust yourself, everything he does makes liberals angry). Amplify things that make him look stupid, make him look inept, make him look corrupt and compromised by establishment ties. And above all, make him look weak. If we can persuade his base to abandon him, he will not last as president.