So on the principle that "if you talk about something 3 times you should probably blog about it", here is my "philosophy of DevRel", what I think the role should be and why.

Five pillars

The role of DevRel has 5 primary facets, but they are not all equally important. They are:

• Tier one:
  • Documentation
  • Engineering
• Tier two:
  • Content & Events
  • Product feedback
• Tier three:
  • Community management

I want to talk about all five, what they mean in practice, and why they sit where they do in the stack-ranking, because there's a lot of nuance in there.

Documentation is the bedrock of DevRel

Everything in DevRel begins and ends with good documentation. The core of the job is, in a very real sense, to get developers to consume the documentation. Documentation (via SEO) is how developers often discover your product in the first place, it's how they learn what it really does and how it does it, it is what convinces them to actually pick up your product and use it for real (or not).

It is also by far the highest-leverage thing you can work on. Every 28 days more than 130,000 people hit the Arize AI documentation sites. Every change you make there affects hundreds of thousands of people eventually, far more than read the average blog post, YouTube video or conference talk. And documentation readers are usually pretty engaged, so these are high-quality interactions. Investment in documentation pays for itself over and over, and very quickly.

I draw a distinction between two types of documentation, both important:

Feature documentation

This is table stakes. Everything your software does, every API, every button, every feature must be documented somewhere so people can find it and figure out how to use it. You may think your product is self-explanatory but you are wrong. Feature documentation has to be comprehensive, readable, and up to date. Machine-generated API docs from your OpenAPI.json are not sufficient, they are the "fuck you" of documentation.

Narrative documentation

If you've ever heard me talk about DevRel then you've heard me talk about the importance of narrative docs. Narrative documentation takes a point of view: it says "I think I know who you are as a user, I think I know what you are trying to accomplish" and it works to get the user there, step by step, as fast as possible. The speed is crucial: users who haven't yet committed to your product are only going to give you a certain amount of their time and attention, and you need to get them from zero to real value as fast as possible.

If you can show users real value quickly, they may be convinced to spend more of their time going deeper. Good narrative documentation encourages this by splitting into phases. Earlier phases get to some basic value, and then later phases build on the work already done to get the user further into the product.

Narrative documentation is a risk: if you pick the wrong user persona, you will spend energy explaining how to do things to people who don't exist and your real users will wander off. So it's important to ground your narrative documentation in data if you have it, basing it off of real numbers of users who have what use case and use other associated tools. And be prepared to explain really basic things: meet users where they are, not where you wish they would be.

You have to do actual engineering

I describe the people I hire as Developer Relations Engineers, and that is a very intentional choice of words: I expect actual engineering to happen, and I expect them to be capable of it. The advent of LLMs has thankfully made knowledge of any particular framework or programming language much less crucial -- a good engineer can vibe-code their way through most stacks with Claude Code's help -- but that requires that they are a good engineer. They need to be able to think about complex problems, break them down, think at a system level, and work within constraints of time and resources. You have to have built a lot of stuff to pick up those skills, so I look for people who have done actual engineering in their past.

What they build is open source software. DevRel is about up-leveling developers into your product as fast as possible and a great way to do that is to throw them a piece of open-source software that builds 50% of what they were going to build anyway, which they can pick up and customize. Again, you need data to make this work: you need to know who your users are and what use-cases they are likely to have (and they're going to have lots, so you're going to need to build lots of different open-source repos, not just one).

Building real software also has other benefits. It's usually a really great way to dog-food a developer product. You're not building a toy, you're building real software that interacts with your product in realistic ways. This is also a great source of product feedback (see below).

And thirdly, open-sourcing real software is a great source of content, which brings us to our third pillar.

Content generation is key, events are mandatory

Documentation is how you make the very best use of a developer's attention. But a big, big part of the job is generating that attention in the first place. This is where your content skills come in.

Content these days is relentlessly multi-channel. We pay attention to five different social networks (Twitter, LinkedIn, BlueSky, Thread and Mastodon) in addition to having a blog, a YouTube channel, and programmatic advertising on TikTok, Instagram and elsewhere.

The good news is that if you have good documentation and are doing real engineering, the first two pillars, then the content writes itself. A good piece of narrative documentation turns very easily into a video walkthrough, a blog post discussion, a Twitter thread or a LinkedIn article, and open-source software likewise gains a lot of attention from the right kind of users. So a lot of the work of generating content is frankly pretty mechanical these days, especially with LLM assistance. But good judgement and human legibility still require human oversight.

Stupid Web Tricks

One type of content that doesn't come directly from docs and engineering is what I refer to as "Stupid Web Tricks": you look at social media, you find what the buzzword of the day is, and you write some content or build a demo that relates to that and garners some attention. If the demo is in some way related to your product, that's even better, but it's not mandatory. If you've built a good content funnel in your documentation, then sometimes it's worth trying to go viral: throw a couple hundred thousand views into your funnel and some percentage of them will convert.

But going viral is not the goal of the team. You can't do it if you don't have good docs and solid engineering to welcome curious people once they've been lured in by your viral content. It's strictly secondary, and it's important to remember that because going viral is fun, gets lots of attention from management, and can therefore be addictive. Be judicious and intentional about it.

Events are mandatory

Some companies think of events as the primary thing DevRel does, but they are firmly in the second tier. This isn't because events don't work: they absolutely do. You get high-bandwidth, real-world conversations with actual users. You get really focused, sustained attention if you give a talk. Events generate leads (more on that later). You have to do them, and they are worth doing.

But events are also expensive. Both in terms of money -- sponsorships cost tens of thousands of dollars before you pile on the cost of flights and hotels -- but also in terms of time. Every day spent at an event talking to 100 people is a day spent ignoring the hundreds of thousands of people who read your documentation. They are a real distraction from the core work. So you have to pick events that work. Those are:

• Big: if there are going to be less than a hundred people, do not go unless it's some kind of dinner event where you get half an hour of their undivided attention, and even then it's a risk.
• Focused: all audiences are not created equal. Are these people likely to want to use your product?
• Value for money: my rule at LlamaIndex was that I was willing to spend about $10 per audience member present. This excluded us from a lot of fancier conferences! Your number might be higher, but you should be thinking about events in terms of dollars-per-head versus, say, programmatic advertising (which is a lot cheaper but also much lower quality attention, so you should use a multiple and not compare them directly).
• Real sources of attention: a booth at a conference is your means, not your end goal. A booth is not going to get much traffic unless you are also speaking at the event. Always try to be a speaker, even if you have to pay to play.

One thing you'll always get at events is great product feedback.

Product feedback

At my last three jobs, Developer Relations has sat in the Product organization, and having looked at how other orgs arrange things it's my firm conviction that that's the second-best place they can be short of being a separate co-equal top-level organization.

Developer Relations folks are second only to sales in getting a firehose of real, actionable product feedback from real users, and are often better equipped to think strategically about that feedback (versus sales who often have short-term pressures to think about). It's important that you put processes in place for that feedback to reach back to the actual Product team. Good Developer Relations people will deliver not just feedback, but product direction and priority: what's wrong, what would fix it, and how important that is relative to other things the Product team could focus on.

Antipatterns for organizational structure of DevRel will put them in Marketing or Engineering. In both those cases they will get measured too narrowly in a way that's detrimental to the team and the company as a whole (see Metrics, below).

Your team should know that their job is not polishing the turd. They should be empowered to imagine how the product can be better, and with processes in place that real change actually happens as a result of their feedback. In today's vibe-coded world they can even build product feature prototypes or commit PRs directly to the product for smaller fixes.

Community management

Community management is undeniably part of DevRel's job but I put it in the third tier by itself, and I do that very deliberately. An organizational antipattern is to treat your DevRel team as a sort of customer support team. That's not what they are. Customer support is an important job of its own that needs different people, different goals and above all different metrics (see below).

Helping users directly is something DevRel should do sparingly. Not because it's not helpful, but because it doesn't scale. Helping one person at a time is right at the other end of the spectrum of documentation, where you can help hundreds of thousands of people at a time.

Instead, DevRel's role in community should be the "management" part. It's about putting systems in place so that users can find each other and help each other without involving you directly. This is way higher leverage and way more effective. Your DevRel team should absolutely be making sure your Discord or Slack community is a safe and productive place, but not usually by helping out people directly.

Metrics

If you've followed all this advice and put all these systems in place the next most obvious question is: how do I know if this is working? And that my friends is the hardest, and from my perspective unsolved, problem in Developer Relations.

There are lots of bad ways to measure DevRel.

If you sit DevRel in the marketing organization they will get measured (inaccurately) by number of dollars of pipeline generated. This is often horribly under-estimated -- somebody reading your documentation might take months to eventually convert, and no cookie lasts that long -- but also driving direct conversions is only one part of the job.

Another bad way emerges if you sit DevRel in the engineering org. Then they'll get measured on features committed, PRs opened, or -- god help you -- lines of code. Don't get me wrong, engineering is part of the job, and those metrics are one thing that can go into measuring developer relations, but they are not and should not be the only way.

The best way I know of is to combine multiple measures. Dollars generated for sure. Features committed (or contributed to the Product team) as well. Attention generated is a third way, measured in views to blog posts, YouTube users, and social media likes.

But a fourth important thing to measure is the vibe. Part of the goal of Developer Relations is, well, relating to developers. They have to like you. You can't measure that with views, or money, or lines of code. Vibes are intangible and immeasurable but keeping the vibes good is part of the job, and a responsible org will know that.

Go forth and relate

I thought this would be a very short post and I was wrong. Turns out I had a lot of thoughts about what this job is and how to measure it, and I hope it's been some use. Now I'll go turn it into a YouTube video, a conference talk, maybe a LinkedIn post...]]> https://seldo.com/posts/do-ai-enabled-companies-need-fewer-people com.seldo/posts/do-ai-enabled-companies-need-fewer-people Sun, 08 Mar 2026 20:34:29 GMT About a year ago I made some predictions about the effect of AI on programming jobs. Block laid off 40% of its staff claiming AI made them more efficient. Is that really true or did they just over-hire? Let's look at some data and see what's really happening.

In February 2026, global venture capital hit a single-month record: $189 billion flowed into startups in 28 days. Three companies got 83% of that funding: OpenAI, Anthropic, and Waymo. Two months into 2026, startups have already raised more than half of what they raised in all of 2025. There is an unprecedented amount of money going into startups, partly because AI is capital-intensive in a way other technology boom cycles have not been. So if you looked at just "headcount per dollar raised" it would definitely look like AI startups are more headcount-efficient. But is that really what's going on?

The average consumer startup raising a seed round in 2024 had somewhere under 3.5 employees, down from 6.4 in 2022. Series A startups that had a median headcount of 57 in 2020 are now at 47. New monthly hires across the startup ecosystem fell by more than 50% between January 2022 and January 2024 according to Carta and its tech-heavy customer base. And tech layoffs, while declining in absolute numbers, continue at a steady clip — over 126,000 workers cut at US tech companies in 2025, with the pace carrying into 2026.

So something real is happening: startups are genuinely smaller than they used to be, even as they're raising bigger seed and series A rounds. They're also hiring more slowly as they grow, and in some cases they're shrinking. It leads to this startling chart:

Is the effect due to AI though, or are we just generally getting more efficient? One way to check would be to see if the effect is more extreme in AI startups. Do we have that data?

There are certainly a lot of AI-powered startups to look at. According to Crunchbase, venture investors poured $425 billion into startups globally in 2025 — a 30% increase over 2024's $328 billion, and the third-highest year on record. AI-related companies captured roughly half of all venture funding, at $211 billion: an 85% jump year-over-year.

And yes, AI-native startups operate with 40% smaller teams than non-AI SaaS companies. AI startups beat regular startups on size of round, revenue per employee, and keeping teams small:

So what does this all mean?

If my prediction last year was true, we would expect there to be a lot of automation going on but also a lot of new tech jobs as demand is unlocked. So far that's not happening; in fact growth has flatlined since 2023, another K-shaped graph:

My read on this is: the startup economy is undergoing a structural transformation here. Startups are substituting compute for labor at an increasing rate. I still remain optimistic that this is going to result in a lot more companies doing a lot more things, but so far it hasn't happened. But companies' claims that they can get by with way fewer people in the age of AI does seem to be true.

* 2026 funding figures annualized from Jan-Feb Crunchbase data. All dollar figures in USD unless noted.

First, I talked about what LLMs are in the first place: really big Markov chains that have hit a threshold where they appear to be reasoning, to a level of fidelity that it makes no sense to argue about whether they are really "thinking" or not. Computers that can reason about the data they are processing is a brand new thing, and it's going to become part of all software.

Then I wrote about what I've learned about writing LLM applications. My key takeaway here is that LLMs are good at transforming text into less text. If you ask them to generate more text than you gave them they are usually pretty bad at that task. But the number of applications where you want to turn a lot of text into less text is truly enormous.

Finally I talked about AI's probably effect on programming jobs, in particular web development jobs because those are the ones I care about most. I think AI-assisted development is going to create a huge amount of new software, which is good, and a whole new breed of highly-assisted software developers, which is also good. I don't think current software developers will lose out on jobs as a result: the demand for software is insatiable and there's more than enough jobs to go around.

On the basis that anything I end up discussing more than three times in real life I should blog about, the next thing I should write about is the current state of the industry and where I think it's about to go. Take this with a heaping spoonful of salt, because I'm not notably good at predictions, but this is what I've got:

My big bet for 2026 is that companies searching for margin and seeing diminishing improvements in the frontier models will start training and fine tuning small models again.

— Laurie Voss (@seldo.com) October 16, 2025 at 5:07 PM

Current state: diminishing returns

The industry right now has a few categories of players that it's worth enumerating:

• Frontier model labs: companies like OpenAI, Anthropic, Google, xAI, Alibaba and DeepSeek are building the very best models that exist, defining what LLMs are capable of, in fierce competition with each other. These models are proprietary and only those companies and their partners can run them.
• Open Source models: some of the same companies above, and also Meta, are releasing open source (or at least open-weights) models that anyone can run. This leads to...
• Inference providers: a whole bunch of companies like Together AI, Replicate, Modal, Fireworks, Groq, BentoML, Koyeb and more that will host the open-source models and run them on your behalf.
• Application companies: an absolute blizzard of companies that are using AI models to build more-or-less domain-specific applications. They can build on top of frontier models, or open source models hosted by the inference providers.

Until recently, any application company wanting to stay ahead of its competitors needed to be building on top of the frontier models. The models were advancing very quickly and all your competitors would switch to them as soon as they became available. This was troublesome for the application companies because the frontier models were expensive, and also since everybody was using the same model it was a big challenge to differentiate your product from your competitors -- UX and prompt engineering were your only, narrow, moats.

My thesis is that that's changing. It's very hard to measure objectively, because the frontier models release benchmarks comparing themselves to each other but change the benchmarks frequently, so there is no single benchmark that I've found that compares, say, GPT 3 to GPT 5's performance. But the "vibe" is that while the jump in performance from GPT 2 to 3 was enormous, the jump from 3 to 4 was less big, and the jump to 5 barely noticeable, with similar progressions for Claude and other frontier models.

Meanwhile, open-weights models are catching up. That's a little easier to demonstrate, as this post and its accompanying graph above show. You can get pretty great results running an open model more cheaply on one of the inference providers than going to a frontier model. If you've got basic inference needs, those models are also getting smaller, which also makes them cheaper:

Next: the search for differentiation (and margins)

In a world where hundreds of application companies are fighting for customers but switching to the latest frontier model no longer brings meaningful differentiation, my thesis is that companies will begin to search for differentiation using fine-tuning.

Fine tuning has always been an available path, of course, but there was no point spending millions creating your fine-tuned model (and hiring the experts to do it) if it was going to be made obsolete months later by the latest frontier release. But things are changing. Fine tuning is getting orders of magnitude cheaper, and services are emerging that will train models for you, meaning you don't necessarily need to hire AI researchers to do it.

There's evidence of this already happening from AirBnB and others, with my favorite example being Cursor: different parts of your interaction are handled by very specific, small, fine-tuned models.

This has two advantages for companies doing it:

1. Differentation: if your model is trained on your data, then (assuming you have more data than your competition) it can be better than your competitors, competing not just on UX and prompt engineering around the same frontier model.
2. Margin: if your models are small, they can be cheaper to run. As the AI bubble deflates revenue and margins will matter more, and companies will be trying to do more with less.

This is already happening, and my is that 2026 will see a lot more of it.

So what should I do?

It's no use making a prediction if it doesn't lead to some kind of action, of course. If you're at a frontier model lab, I have no advice for you other than "get money off the table while your valuation is still insane". If you're at an inference provider, I think you're in for a good year. If you're at an application company, I think now's the time to start building up your dataset and looking at smaller models.]]> https://seldo.com/posts/ai-effect-on-programming-jobs com.seldo/posts/ai-effect-on-programming-jobs Mon, 17 Mar 2025 00:32:34 GMT There's been a whole lot of discussion recently about the impact of AI on the market for web developers, for programmers in general, and even more generally the entire labor market. I find myself making the same points over and over, and whenever I do that it's time to write a blog post about it, so this is that.

Doom and utopia are not our only options

There are two extreme takes on the impact of AI on programmer jobs:

1. AI will take all programming jobs (usually advanced by people selling AI)
2. AI will not take anyone's job (usually advanced by grumpy older developers, which I usually am)

I would like to advance a third option, which is that AI will create many, many more programmers, and new programming jobs will look different. Here's why I believe this:

1. There is far, far more need for software than there are programmers to build it. You can see this in the astronomical salaries for programmers, with senior engineers at big companies making millions of dollars every year.
2. AI does not eliminate the need for structured thinking about how to solve problems, which is the core of programming. It presents a new layer of abstraction. This is a point I have been making since 2018, and in 2020 and in 2023: new layers of abstraction create new kinds of programmers.

AI allows software development without knowing programming

AI has created a very powerful new abstraction that can potentially remove the need to know a programming language from the job of being a developer. Karpathy calls this "vibe coding". I haven't thought of a name I really like for this new type of developer yet. "AI engineer" is taken to mean a different thing. AI Composer? Software synthesist? In a 2012 post I imagined them as blue collar knowledge workers but that's a problematic label on lots of levels.

These new developers will know how to think in a structured way, they will know principles of how software should function architecturally, but they will know less about programming language constructs and syntax. They will be architects more than developers, instructing AI to put together software for them without needing to know about "low-level" implementation, by which I mean the code. The same way current coders don't need to think about how assembly code works. Maybe "AI architect"? Including "AI" in the name feels wrong though, it's got a horseless carriage feel. All programming will involve AI, so including it in the name will be redundant.

New abstractions are always showing up

The important thing to note is that this is not some unprecedented change. We've been adding levels of abstraction to programming since it was invented. A few years back I tried to visualize "the stack" we talk about when we talk about "full stack" developers and came up with this vibes-based chart, which I've updated to account for the last 5 years:

The ability to build software with AI is crushing down some previously very durable parts of the stack, like HTML itself, and replacing that expertise with the ability to effectively prompt and coordinate AI.

More abstraction == more software

This new layer of abstraction will do what all the others did, which is improve the speed at which we can produce software of a given quality. Assuming AI lets us build software of equal quality much faster than we did before, this will produce either

1. higher quality software, produced in the same amount of time
2. more software of the same quality
3. enormously more software of lower quality

I think we will see all three at the same time. Some AI-assisted software development will raise the bar for quality to previously cost-ineffective heights. Some AI-driven software will be the bare minimum, put together by people who could never have successfully written software before. And a great deal will be software of roughly the quality we see already, produced in less time, and in correspondingly greater quantity.

Don't worry about your job

So if software development gets rapidly cheaper, will programmer salaries fall? I'm not sure that they will. As I said earlier, there is a vast amount of currently pent-up demand: software that needs to be developed that the people who need it can't afford. You see this every time you visit a shitty website for your doctor or dentist or insurance company or random department of government. It's not that we don't know how to build better software, it's just those organizations don't want to pay for it at current prices. Lowering the price of software deveopment will also induce demand for even more: software for people too poor to afford it previously, software for niche audiences too small to sustain it before, will suddenly be huge markets.

Instead I think we'll see lots more software developers. This vast wave of new developers will probably earn, on average, less than the programmers of today. But the programmers of today will still be employed. Existing programmers are not, mostly, going to be fighting with new developers for jobs. They'll keep their jobs (and their salaries), and a whole bunch of new, cheaper jobs will also be created, unlocking a long tail of software that currently just goes unwritten. Some, producing software of previously-unimagined quality, will even earn more than before. It will look kind of like this:

Don't worry about the fundamentals, whatever those are

Will this mean that new programmers don't learn the fundamentals? No. New programmers, like every programmer since the dawn of programming, will learn what they need to get the job done. They'll start at the top of the stack, writing with prompts only, and they'll slowly pick up knowledge lower and lower in the stack, as and when they need to learn it. This will be accompanied, as always, by a great deal of whining from people who didn't have the advantage of these levels of abstraction when they learned to build software, and as usual the whiners can be safely ignored.

Having people writing at higher levels than today will not lead to forgetting anything. There are still people who write assembly today; there are just way, way more programming jobs higher up the stack. It will not create a crisis for new grads, either. They will get jobs writing AI-assisted code and work their way down the stack (and up the salary bands) as they gain experience. Don't worry about them, they'll learn this stuff eventually. Or they won't need to, and that's fine too.

Say hello to the software developer

The net result of all of this for the programming market is: more software, better software, more programmers, better programmers. But the adjustment won't be without pain: some shitty software will get shipped before we figure out how to put guardrails around AI-driven development. Some programmers who are currently shipping mediocre software will find themselves replaced by newer, faster, AI-assisted developers before they manage to learn AI tools themselves. Everyone will have a lot of learning to do. But what else is new? Software development has always evolved rapidly. Embrace change, and you'll be fine.

And I think I've decided on a name for this new type of AI-assisted developer: we can call them software developers. Giving them some other name will create gatekeeping where none needs to exist, it will falsely imply they can never move down the stack. I've been in this game too long to fall into that trap. They develop software, so they are software developers. That's the only qualification they need.]]> https://seldo.com/posts/what-ive-learned-about-writing-ai-apps-so-far com.seldo/posts/what-ive-learned-about-writing-ai-apps-so-far Mon, 20 Jan 2025 22:10:58 GMT I started writing a post called "how to write AI apps" but it was over-reach so I scaled it back to this. Who am I to tell you how to write anything? But here's what I'll be applying to my own writing of AI-powered apps, specifically LLM applications.

A battle I've already lost is that we shouldn't call LLMs "AI" at all; they are machine learning and not the general intelligence that is implied to the layman by the name. It is an even less helpful name than "serverless", my previous candidate for worst technology name. But alas, we're calling LLMs AI and any parts of the field that are not LLMs are being drowned out by the noise around LLMs. I do this a lot at my day job; I am certainly part of the problem.

But if we can't call it what it is we can at least know, as practitioners, what it is: very fancy autocomplete. At scale, autocomplete can perform tasks that look like reasoning, and at some level "looks like reasoning" and "is reasoning" have blurred boundaries. Nevertheless, as the inventor of software you should be clear-eyed about the limitations of these systems, and not try to get them to do things they can't do.

LLMs are good at transforming text into less text

This is the biggest and most fundamental thing about LLMs, and a great rule of thumb for what's going to be an effective LLM application. Is what you're doing taking a large amount of text and asking the LLM to convert it into a smaller amount of text? Then it's probably going to be great at it. If you're asking it to convert into a roughly equal amount of text it will be so-so. If you're asking it to create more text than you gave it, forget about it. The rest of this post is really just examples of this rule in action.

LLMs only reliably know what you just told them, don't rely on training data

LLMs are trained on gigantic quantities of information found on the Internet. As a result, tools like Claude and ChatGPT can answer various general-knowledge questions. This creates a huge temptation to create an app that uses what the LLM already knows to perform some task. This is never effective. You don't know what the LLM has been trained on (famously, because it's often illegally acquired, so those training them refuse to say), therefore you don't know the limits of its knowledge or when it will start to hallucinate.

The way to get around this is to give it the answers. Want to know what a contract says? Give it the contract. Want to know what a video is about? Give it the transcript of the video. Want it to make a decision? Give it all the same information you would use to make that decision. These are all just turning text into less text. It's great at that. (Yes, you can try and get around LLMs only knowing specifically what you just told them by fine-tuning your LLM. Good luck with that.)

This is why Retrieval-Augmented Generation (RAG) is not going anywhere. RAG is basically the practice of telling the LLM what it needs to know and then immediately asking it for that information back in condensed form. LLMs are great at it, which is why RAG is so popular.

LLMs cannot write for you

This is firmly in the "less text into more text" category. If you give an LLM a prompt, it can spew out a novel-sized block of text if you ask it to. But LLMs only know what you tell them. Give it a short prompt and ask for long text and you will get endless waffle, drivel, pointless rambling, and hallucinations. There is no way to get an LLM to perform the thought necessary to write something for you. You have to do the thinking. To get an LLM to write something good you have to give it a prompt so long you might as well have just written the thing yourself.

Let them self-correct, multiple times if necessary

A wild thing about LLMs is that they can observe what they've done and decide whether they've done a good job. This is sometimes called self-reflection and it's a key part of what agents do, and I can't emphasize enough what a good idea it is to give your LLM the chance to figure out if it fucked up, and a chance to try again. It adds complexity to your app but it will pay you back in reliability many times over. LLMs are bad at one-shotting but if you give them a couple of swings they often get it. It's both the curse and the magic of them being nondeterministic.

Have the LLM do as little as possible

Per the above about reliability: you know what's really reliable? Regular programming. It takes inputs and turns them into outputs, the same way every time, according to extremely precise instructions. If there is anything you are asking the LLM to do that could be accomplished by writing some regular code, write that code. It will be faster, cheaper, and way more reliable to run. LLMs are capable of handling ambiguity and complexity, and it's amazing, but the less of it you give them to handle the better they're going to do. Regular, declarative programming can work wonders and you should use it.

LLMs can help a human perform tasks, they cannot replace a human

This is really a corollary of all of the above. If you have a really great prompt containing lots of careful instructions, and provide all the data needed to perform that task, plus lots of chances to reflect and try again, with as much regular code as possible, LLMs are going to be able to perform that task. If you have a whole lot of these collections of prompts and data and code you can create an agent that can perform *lots* of tasks. At that point, it's tempting to look at somebody's whole job and say "this job is really just these 20 tasks, I have created an agent that can do all of these tasks, therefore I can replace this person". It's tempting but I have never, ever seen it work.

Jobs are more than collections of tasks. Jobs require prioritization, judgement of exceptional situations, the ability to communicate ad-hoc with other sources of information like colleagues or regulations, the ability to react to entirely unforseen circumstances, and a whole lot of experience. As I said, LLMs can deal with a certain amount of ambiguity and complexity, but the less the better. Giving them a whole, human-sized job is way more ambiguity and complexity than they can handle. It's asking them to turn text into *more* text. It's not going to work.

It's easy to argue that as LLMs get bigger context windows and agent tools get more sophisticated that the ability to replace a whole human will show up: after all, LLMs are good at turning text into less text. How much text do you need to replicate everything a human knows? I don't know but it's a lot more than any LLM can handle right now. In the meantime, your attempts to replace humans with LLMs are going to suck. Let your app augment and accelerate a human, I've seen that work lots of times and be very effective.

In particular, because I see this over and over at hackathons: please stop trying to write an LLM app that replaces a doctor or a lawyer. The LLM does not have a law degree or a medical degree baked into its training, and you definitely cannot fit 7 years of medical training into even the biggest prompt. I don't think you can reliably get an LLM to replace any human but especially not a doctor, do not trust your health to autocomplete that is just trying to be helpful. Do not get sued into oblivion because you ChatGPTed your legal terms.

LLMs are awesome and limited

Depending how much of the hype around AI you've taken on board, the idea that they "take text and turn it into less text" might seem gigantic back-pedal away from previous claims of what AI can do. But taking text and turning it into less text is still an enormous field of endeavour, and a huge market. It's still very exciting, all the more exciting because it's got clear boundaries and isn't hype-driven over-reaching, or dependent on LLMs overnight becoming way better than they currently are. Take a look at the possibilities, find something that fits within these boundaries, and then have fun with it. ]]> https://seldo.com/posts/ai-ml-llms-and-the-future-of-software com.seldo/posts/ai-ml-llms-and-the-future-of-software Fri, 15 Sep 2023 20:17:08 GMT I recently left my job at Netlify and have been looking at what's next. At Netlify I became very interested in the power of ML, AI and LLMs in particular, and that's the area I've been looking in. But there's a lot of hype and buzzwords around, so I wanted an explainer I can point people to when they ask "but what is all that stuff really?" It's intended to be short, and so it simplifies a lot on purpose.

AI is (now) a marketing term

When I went to school for computer science 20 years ago, Artificial Intelligence (AI) meant a specific thing: computers that could perform symbolic reasoning about the world, the way humans do. That's now referred to as AGI, or Artificial General Intelligence. Nobody has invented AGI yet; there are still no computers that can think and have an identity the way humans do, although some people suspect we are getting close to that and will be there soon (I'm skeptical).

What there has been is a huge series of leaps forward recently in Machine Learning, or ML. Companies using this tech have tended to call themselves "AI" companies, which is true in that ML is a subset of AI. It's like if companies that made bicycles called themselves transportation companies. But what's ML?

ML is learning from examples

Machine learning involves giving a computer a huge number of examples of something and letting it learn from those examples. For instance, you can give a computer a million pictures of words and tell it what those words are, and thus train it to recognize letters and convert pictures of words into text. If you have an iPhone, that's a feature now built into your photos app: you can copy and paste text out of pictures. Another example is email spam filters: if you give a computer a million examples of emails and label them as spam or not-spam, it gets pretty good at figuring out what spam looks like. You use ML every day.

Because it has a lot of useful applications like those, ML in general has been pretty hot for a few years now. But where the real leap forward has been in the last 12 months has been in the area of Large Language Models, or LLMs.

LLMs are really big Markov chains

If you know what a Markov chain is, it's easy to think of an LLM as a really big one of those, but don't worry, I'm not going to assume you do.

Imagine you made a computer read every book in the world, and then got it to construct a list of every three-word phrase in those books. Then for each phrase, you got it to make a list of all the words it saw come after that phrase and rank them by how often that happened. Then you gave it the phrase

The cat sat on the _______

and then you asked it to predict what word would come next, based on the list it had made. It would almost certainly predict "mat", because in all the English language in the world that's how that phrase is most often completed. Similarly, if you gave it

The capital of the United States is

it would probably give you "Washington, D.C." because again, that's just a really common phrase. But already something interesting is happening, because it looks like the computer "knows" something: the capital. But it doesn't, it's just doing word completion.

Now scale up

Now take this sentence-completion and scale it up, way up, by factors of millions and you get ChatGPT, the LLM that everyone's been talking about. You can give ChatGPT a question, and it will give you what appears to be an answer, like so:

It not only "knows" that the word should be "mat", but it gives you context about why. But this isn't doing anything fundamentally different. It gives an answer because statistically, answers are what usually follow questions. It gives the context because that context is what often surrounds that phrase. It's still just doing math, and not even very complicated math, just at enormously high scale. It doesn't actually "know" anything.

A quantitative change becomes qualitative

And yet something magical has happened. At some point what it does changed from looking like sentence completion to looking like understanding. What qualifies as understanding things is a philosophical question and a hotly debated one at that, and there are some people who think that computers are beginning to show real understanding here, but most people are on the side that this is still just math.

But whether it's real or just a convincing simulation doesn't really matter: at some point looking like understanding is nearly as useful as truly understanding it.

Giving computers context

Computers until now have been very good at processing data: they can stack up huge piles of data, sort it, filter it, move it around, spit it back out, all at enormous scale and impressive speed. But what they haven't been able to do is turn data into information. They haven't been able to understand what they're working with. But now, if you give a computer context, it can appear to reason about that information:

It figures out a conclusion from a related set of facts, namely how old Susan is. What's really interesting here is that this set of facts are not part of the set it was trained on. When it read all the books in the world, Susan's age wasn't in there. In fact, I never told it Susan's age. At some point along the line, prediction of the next most likely word became reasoning. This is the surprising and frankly spooky outcome that has made LLMs so interesting to me and many other people.

Computers that understand changes everything

There are a bunch of really neat things you can do with this new-found ability of computers to read information and understand it. Give it your technical documentation, and then ask it questions about how to do things. Give it a pile of legal contracts and get it to tell you what they do and don't allow you to do. Give it a pile of research papers and get it to summarize them and find connections between them. Give it access to your company Slack and get it to summarize everything that happened in the last two weeks while you were out on vacation.

LLMs are going to end up in everything

This transition of software from being a tool that we use to a sort of low-level assistant that we can partner with is going to change every piece of software out there. Software that doesn't have the ability to understand what it's working with is going to begin to feel strangely limited, even broken. It's a very exciting time to be working in software, and it's why it's what's next for me.]]> https://seldo.com/posts/theres-no-such-thing-as-the-fundamentals com.seldo/posts/theres-no-such-thing-as-the-fundamentals Sun, 19 Mar 2023 20:12:35 GMT Nine months ago I gave a talk about how there is no such thing as the fundamentals of web development. It's a thing I have been saying for years but keep getting pushback on, and when that happens it's time to write a proper blog post about it rather than just arguing with people one at a time.

What is a fundamental?

If I'm going to argue that there's no such thing as the fundamentals of web development, there's two ways you can disagree with me: you can disagree with me about what "a fundamental" is, or you can disagree about what web development is. I apologize for reaching for a dictionary definition of fundamental, but it is "forming a necessary base or core; of central importance" and the key point of disagreement I have with other people here is the stipulation that it be necessary.

There are lots of things that are valuable to know. There are lots of things that are important to understand. There are lots of things that are best practices. But a skill can be all of those things without being necessary. Fundamental to driving a car is understanding how to operate it: what the accelerator does, what the brakes do, and what effect turning the steering wheel will have. Important, but not fundamental, is knowing traffic laws. You can operate a car just fine without knowing how to operate it safely, as anyone who's ever driven in traffic will attest. So obvious is it that knowing traffic laws isn't fundamental to driving that you can arrive in a foreign country and rent a car without having taken the local driving test, you literally don't know what the laws are and yet away you go.

Also not fundamental is understanding how brakes and accelerators and steering wheels work: it can be helpful to know about motors and gears and brake discs and whatnot, but you don't need to understand them to operate the car. They are therefore not, by definition, fundamental because they are not necessary.

That doesn't mean knowing how a car works and knowing what the traffic laws are isn't useful, or not important, or not a good idea. You'll certainly drive better if you do. But they're not necessary, and therefore they aren't fundamental. If you disagree with this definition you're free to claim things are fundamental but dictionaries are not on your side.

So that leaves the remaining possible source of disagreement:

What is web development?

If you say that "putting together HTML, CSS and JavaScript" is the definition of web development then, tautologically, there are some fundamentals of web development: HTML, CSS and JavaScript. But I have been a web developer since before CSS and JavaScript were invented, and I promise I was still making web pages then, so I don't buy a definition that includes any specific technology. When I started web development, knowing how a physical web server worked was essential to getting a website online. Knowing how HTTP worked at the protocol level was also considered vital, because you were going to have to write code to parse those headers yourself. What's necessary to put together a website today is not the same as what's going to be necessary to put together a web site tomorrow. So if those skills change, are they really fundamental? My position is that they are not. So then what is web development?

My working definition is that web development is creating a site that can be viewed in a web browser. So that software I wrote to parse HTTP headers and spit out plain HTML? Web development. That PHP I wrote? Web development. That Ruby I wrote? Still web development. The JavaScript and CSS? Also web development. But likewise, somebody who is configuring WordPress and installing a pile of plugins to get a website out to the world? They are also a web developer. Somebody who draws a website in Webflow, or customizes a template on SquareSpace? Also a web developer. Web development is the product, not the method.

"But $skill is fundamental!"

It's at this point in the conversation that somebody will haughtily tell me that you can't be a web developer if you don't know how some specific thing works, and all I can say is that's bullshit. Whatever skill it is you've mentioned -- is it JavaScript? CSS? HTML? Design? Deployment? Testing? Performance? Responsiveness? Accessibility? -- I can point to thousands upon thousands of working web developers who don't know that thing and yet are getting web pages out into the world. They're often terrible web pages, user-hostile nightmares, especially for those who discard accessibility, and yet there they are, out in the world, existing. The developers who don't know whatever pet skill it is you think is fundamental exist whether you respect them or not, but I respect them. They're doing their best, and like all of us they will slowly get better.

"If you only learn a framework, you'll never be a real web developer"

This is just the same as the last argument. Says who? If somebody only knows React or only knows Rails or only knows PHP they are still getting web pages out into the world, they are still a web developer. Without learning more will they sometimes make bad designs, or run into problems they can't solve? Sure. But that doesn't make them not a web developer.

"When you learn web development, you should start with $skill"

The answer is still no. The place to start in web development is whatever lets you get a website up. Where you started when you learned web development has no special significance that others should be required to learn it first.

Gatekeeping is for the weak

"Okay, but you're not a real web developer unless..." Stop. As long as a web page exists at the end of your work, you're a real web developer. Gatekeeping is a mechanism used by the weak to reinforce their power, by excluding others, and I'm not here for it. Web development is a broad, deep field and there is room for an enormous diversity of developers who know different things and work different ways.

"Eventually you'll have to learn $concept, so it's fundamental"

Again, whatever concept it is you think you've found that every web developer learns eventually, I am happy to point you to somebody who's been happily churning out web pages for years unaware of that concept. Web technologies are incredibly powerful and flexible, and you can get enormous amounts done without venturing up or down the stack from where you started. You can often certainly get more done if you go up or down the web technology stack. There will certainly be some things you can't do unless you climb down to a certain level of the stack. But not every web developer needs to be able to make a web page that can do everything a website can do, in fact there are hardly any who can claim that because there's just so much to know.

"But accessibility!"

A strong pushback I get in my mission to prove that nothing is fundamental to web development is the argument that understanding of accessibility is necessary, and that treating accessibility as "optional" is malpractice (some people also think this about performance, or mobile responsiveness). But you don't have to understand accessibility to produce accessible web pages. If you just write plain HTML it's accessible by default, whether or not you've heard of accessibility. If you're using a good component library, your components will likewise be accessible without you needing to understand accessibility. The existence of lots of terrible web pages put together in frameworks that ignore accessibility doesn't mean frameworks can't get accessibility right without intervention.

"Well they have to know HTML at least"

Do they? This is usually the last stop on somebody's tour of things they think people need to know, after I've shown them sites put together by people who've never heard of HTTP, don't know what a form is, don't care about JavaScript, or style things without any knowledge of CSS. HTML, foundation of the web... surely, surely HTML is a fundamental?

But it's not. If I can draw a web page in a GUI tool without knowing HTML, if I can install WordPress plugins that do what I need without knowing HTML, then I can get web pages out into the world without knowing HTML. It's useful, it's important, it's powerful, but it's not necessary. It's already been unnecessary for years, and as tooling continues to evolve and increase in power it will become ever less necessary. Will it always be useful to understand HTML even if you're not hand-coding it? Probably, just like it will always be useful to understand HTTP status codes and what they mean. But not essential. Not necessary. HTML, much as I love it, is just as un-fundamental as every other thing.

Embrace the chaos

If you've spent a career becoming an expert in some technology, being told that technology isn't fundamental can be discouraging. It can sound like I'm saying your skills aren't important, or valuable. Of course they are, and you don't need me to prove that. Are you using your skills to make web pages? Then they are vital to your type of web development, and they're good skills and you should be proud of them. But you don't need to draw a line in the sand that says everybody else has to have your skills.

The joy and the power of web development, since its inception, has been how accessible it is, in the sense of accessible to new developers. It's always been the case that you can learn enough in an afternoon of messing around to get a web page (a terrible, messy web page) out into the world. Getting a web page up is a great feeling, it's a rush of power and possibility that creates the feedback loop of satisfaction that has us all learning more and building better every single day. Drawing lines in the sand, getting cranky about people who don't know some particular thing, does nothing to help you or them experience that joy. If it's important for them, they'll get there eventually, and if they don't, you don't need to worry about it, they're still a web developer and so are you.

Web development is fun, it's fast, it's incredibly powerful, and it's constantly evolving. A bigger web is a better web, and the way we get there is by welcoming everyone who wants to play, no matter what they learn first or how they get the job done. There's no such thing as the fundamentals, and that's fine.]]> https://seldo.com/posts/the_case_for_frameworks com.seldo/posts/the_case_for_frameworks Fri, 10 Feb 2023 00:00:00 GMT Today I read Alex Russell's post The Market for Lemons and I found myself compelled to write a rebuttal. I am a big fan of Alex's work in general but not of this post in particular, which is very long, so allow me to attempt to summarize it:

• JavaScript-heavy single page apps (SPAs) are very popular
• The web is mobile-first and Android-dominated
• JavaScript-heavy apps do not perform well on mobile Android devices
• We were sold on said JavaScript frameworks by a few noisy individuals
• They were lying when they said the frameworks were good (the specific analogy drawn is: like oil companies denying climate change)
• This is terrible, and so are those people
• We must kick JavaScript frameworks to the curb (specifically: we must throw out both the baby and the bath water)

This is a song Alex has been singing for a long time and I'm on board with caring about performance and making the web more accessible to more people (especially poorer people with slower devices and worse internet connections). But this post goes way too far in ascribing malice aforethought when the explanation, in my opinion, is an economic one. As is too often the case when Alex gets passionate I feel like he ends up calling developers stupid for being fooled into using frameworks.

Disclaimer: I was involved in npm, the world's repository of open source JavaScript software. I quite like JavaScript, and as a mediocre web developer of many years I am a fan of frameworks (we'll get into why shortly), and dislike being called stupid. Take all of this bias into account when reading the rest of this.

Hawks and Doves

To understand my explanation I'm afraid I have to delve into evolutionary game theory just for a second. You can click through to Wikipedia for a fuller description but simplifying greatly the Hawk Dove game, first posited by Maynard Smith, imagines a finite world with limited resources and two species. One species, called the hawks, will fight for resources. The other species, called the doves, will share resources. When either species gets resources, they breed, creating more of themselves.

A key finding of the game (which you can simulate in software to varying degrees of complexity) is that neither species can "win". In a population of entirely doves, a single hawk will fight doves and get way more resources, producing more hawks. In a population of almost entirely hawks, two doves will cooperate and do better than the hawks, which waste resources fighting, and there will be more doves soon.

In fact the finding is that you can throw in any number of hawks and doves to start with but for a given quantity of resources you will always end up at the same mix of hawks and doves, a point of equilibrium known as an evolutionarily stable strategy.

All you need to take away from this to understand the argument I'm about to make is that: while a world of doves all cooperating with each other would be nicer, indeed the best possible world, it is not the world you end up with because it is not stable. It only takes a single hawk to show up to throw it out of whack.

How this translates to the world of software is: it is my assertion that the world as it exists is a world in a relatively stable equilibrium. Reality is complicated, so there aren't just two teams of hawks and doves, there are tens of millions of software developers out there, all working relatively independently and in their own best interests with different priorities and resources and trade-offs. The result is the world we see, and despite it being not the best possible world depending on your own priorities, the equilibrium exists for completely rational reasons.

So: there is no secret cabal of charismatic influencers destroying what would be a perfect world without them. We cannot blame anybody for the state of things. We created this world ourselves, collectively. If we want to change the status quo, we need to understand the rational forces of self-interest that created it. We have to change the game, not just yell at the players.

Let's talk about React

In our world of rational actors in a stable equilibrium 71% of devs say they are using React for some or most projects and that number has been rising every year for 9 years now, which is when I started tracking these things. So let's do two things: first let's stop talking about "JS frameworks" in the abstract: all the frameworks are different, they make different trade-offs and have different performance characteristics. Let's instead talk concretely about React, the elephant in the room. And the second thing we should do is assume there's a good reason, and not that developers being bamboozled by liars. What could it be? The answer lies in the economics, I think.

React is popular despite being bad for performance

I'm absolutely not going to claim that JavaScript-heavy single-page apps are good for performance because they're not. They are slow to load, slow to render, hog resources and are frequently more fragile than vanilla HTML and CSS would be. Developers of JS frameworks are, I think, pretty open about these limitations, which is part of why server-side rendering is such an important feature in modern frameworks. Most React proponents will tell you that not every website needs to be a React app. I also think developers aren't stupid, so they know that they are making these trade-offs when they pick React. So why do they?

Single page apps are pretty useful, actually

Alex makes the astonishing assertion that "vanishingly few scaled sites feature long sessions and login-gated content". The above, as an anecdote to the contrary, is a screenshot of my start screen on my personal laptop in Firefox; it is roughly speaking a list of my most-visited websites. Ignoring Alpaca, an unreleased side project of mine, all but one are SPAs. Taking the requirements of "long sessions, login gated content", all but one should be single page apps.

App	Is SPA	Should be
Tumblr	Yes	Yes
Instagram	Yes	Yes
Chase	Yes	Yes
GMail	Yes	Yes
Zillow	Yes	Yes
Pinafore	Yes	Yes
LinkedIn	Yes	Yes
Google Maps	Yes	Yes
Google Sheets	Yes	Yes
Slides.com	Yes	Yes
Netlify	Yes	Yes
Questionable Content	No	No
Google Drive	Yes	Yes
Mode Analytics	Yes	Yes
Bamboo HR	Yes	Yes

This is just one data point and proves nothing. Instead take a look at the top 100 sites by traffic and consider how many have long sessions and login-gated content. Certainly the long tail of websites in the world is probably mostly static, but the big sites -- where all the development dollars are spent -- are overwhelmingly rich, complex sites. The money is in rich apps, not because the money is being wasted, but because the content and services people pay for are highly personalized and dynamic, where rich apps shine.

React saves developer time

This is sometimes described as "developer experience" but from an economic standpoint, money doesn't care how the developer experiences anything, money cares how fast they work. Any framework obviously takes more time to get started than a simple static site, but I think it's uncontroversial to claim that over time, a framework like React is going to save your developers time: that's what frameworks are invented to do. Let's be clear why that is, though.

React has already been used to solve your problem

This is the primary technical benefit of any framework: it has been used many times to solve problems of great complexity, usually greater than anything you need to do. So you're not asking "is there a way to do this?" you're asking "what is the way to do this?" And more often than not, the solution is easy to find because somebody has written documentation, or a blog post, or an open-source component that does it. Frameworks have compounding returns this way: the more people who use them, the better they get.

That network effect is part of why React, regardless of its technical merits, is currently the best choice for a host of basic applications: the React ecosystem will get you 80% of the way there with pre-built solutions and well-documented boilerplate. You can concentrate on the things that make your app different. And this brings us to another big reason.

You can hire people who already know React

When 71% of the developer population is already using React, hiring for React developers becomes easier, and from an economic standpoint that is a huge advantage. Finding developers is expensive, on-boarding developers is expensive, retaining developers is expensive. A React developer is easier to find, a developer who already knows React requires less on-boarding, and a developer using a framework they enjoy (and they know they can use elsewhere) is happier, so React developers are easier to retain.

React's performance hit loses customers

But all of this developer time saved and convenience comes at a cost: performance. A React app is, as Alex never tires of pointing out, problematic at best on mobile and even more so on a low-power Android device. At the margins, the effect is going to be lost customers. People will either be unable to use a site at all, or find the experience so painful that they'll churn. This is bad! Losing customers is always bad... right?

Developer time is more valuable than some customers

This brings us back to the Hawk Dove game. Our hawks are developers, greedily sucking up browser resources to save themselves time. The doves are customers, meekly trying to load a website and hoping it will run. If the developers were allowed to use all the resources, the site would run for nobody -- an all hawk population. But on the other hand, if the website were designed to run perfectly for every single customer, in every location, on every device, then the developers would spend their entire time in optimization work and never ship anything new.

The status quo is an equilibrium between developer and customer experience

Which brings us to the reality. Developers are expensive. Their salaries are paid, ultimately, by customers. But not all customers are equally valuable. The less money a customer is likely to pay, the more likely it is to be worth to sacrifice their experience in favor of quickly shipping some feature that will, at the margins, attract or retain some other, wealthier customer.

The brutal truth is that the status quo reflects the economics of the developer market: expensive developers mean cutting off poorer customers. The observed reality is that modern web development sacrifices the experience of poorer people, and the model we've laid out here explains why that is: X > Y, where X is "cost of developer time" and Y is "cost of customers lost".

Shifting the point of equilibrium

But this is bad! Here Alex and I have always agreed. We want an open, accessible, world wide web. We want as many people as possible to have a great experience. But simply saying it's bad won't help. To shift the equilibrium, we need to change something about the equation. The variables I can see are:

• The cost of developer time (we'd want it to go down)
• The amount of developer time required (also better if lower)
• The value of the customers lost (if higher, this would shift to losing fewer of them)

How do we tackle this? Is there something that makes developers cheaper? Is there something that makes developers faster? Is there something that makes customers richer?

The solution is more frameworks, better frameworks

The answer is frameworks. They're not solving the problem perfectly, but the nature of the Hawk Dove game is that it's impossible to get the ideal world. But making frameworks better works on all three fronts:

• Making frameworks easier to learn means more people can become developers, and increasing the supply of developers will make development cheaper
• Making frameworks richer means more of the work is done by the framework, so you need less time to build the same level of functionality
• Making frameworks technically better pushes the equilibrium as well, by giving developers better efficiency for "free", thus expanding the number of customers who can access the site on the same devices. Relatively speaking, the customers get "richer".

And that's why frameworks are the dominant development paradigm. They don't work against users, making their experience bad. Economics is what makes their experience bad. Frameworks in their frantic, endlessly competitive push to make developers faster, better and more numerous are working tirelessly on the behalf of users, making their experience better, and making developers better too.

Why not a more efficient framework than React?

This brings us to the final question: if we're sold that frameworks in general are a good thing, why is React in particular dominant right now? There are delightfully light and performant frameworks and Alex lists several of them in his many extensive footnotes. Alex isn't against all frameworks, just the big heavy ones, React being overwhelmingly dominant and Next.js being the biggest framework within React.

But I think we've already answered the question. React has huge network effects: a long history, many solved edge-cases, a huge developer base, an enormous ecosystem of drop-in solutions. If you want to get rid of React's inefficiency you can't just be more efficient (because of Hawks and Doves, it just doesn't matter enough). You have to be better at something that makes developers faster, and it has to be a LOT faster to overcome the network effects. No framework has hit that threshold yet, but that doesn't mean it won't happen.

You're doing just fine

In the meantime, my fellow developers, don't feel stupid. You have not had the wool pulled over your eyes by a cabal of fast talkers. You've been making sensible decisions in the best interests of yourself and your users in the aggregate, and you don't need to feel like you're a bad person for using the popular frameworks. They are popular because they are a damn good idea.]]> https://seldo.com/posts/crypto-the-good-the-bad-and-the-ugly com.seldo/posts/crypto-the-good-the-bad-and-the-ugly Thu, 06 Jan 2022 02:44:36 GMT It has been very frustrating watching the rise of cryptocurrency (which, forgive me cryptographers, I will be calling "crypto" from here on) because a whole bunch of smart people in tech seem to be very, very excited about it. When good new things show up in tech, I've generally found them intriguing. I'm by no means an early adopter, but once the train is leaving the station I am generally on board.

But crypto hasn't grabbed me like that. Every time I dig into crypto I find things that seem stupid, or useless, or actively bad. But so many people are into it! I'm a big fan of the wisdom of crowds, especially when it comes to technical choices, so a big crowd of people doing something that seems stupid really eats at me. I must be missing something!

So here, mostly to help me think about it for myself, I'm going to examine all the things that are good about crypto, and then all the things that are bad.

The good

Technical achievements

I'm going to assume you are already familiar with the basics of what a cryptocurrency is and how it works. Blockchain-based cryptocurrency allows you to create basically one thing: a currency, and a set of rules that govern that currency. You define how transactions in the currency happen, and you can define side effects to transactions. In a lot of the coins, like Bitcoin, the rules and side effects are very simple, and the result is to create something that works like money. You can store it, you can trade it.

In the more extreme variations, like Ethereum "smart contracts", the side effects can be "execute this piece of code", which allows you to create anything from a money laundering machine to things like Distributed Autonomous Organizations (DAOs), which are essentially wallets that hold money and have their own set of rules about how and why money should go in and out of the wallet. Those rules can involve holders of tokens who have votes in proportion to how many tokens they have, or any other set of rules about the governance of that organization. But they are still, at heart, money plus rules about money.

This is all quite cool and impressive, technically. A whole bunch of really tricky technical problems had to be conquered to achieve the goal of having a global distributed system on which people can execute arbitrary code. It's neat! As I see it, crypto represents two major technical accomplishments.

First, a system where nodes can cooperate without trusting each other, certainly a hard problem, has been solved. The solution was the currency itself: by cooperating, you earn money in proportion to how much you've contributed to the system. If you don't cooperate you don't get paid, so the incentive is to cooperate. You don't need to trust them.

Second, more important for the crypto with things like smart contracts, is that the problem of abuse has been solved. In a network of machines that allows anybody to execute arbitrary code, a huge problem is going to be bad programs running wild and eating all the resources of the network. That problem has been solved by making every action cost money (or computing time, which is essentially a proxy for money). Over-use of network resources is solved by making it prohibitively expensive.

The fact that any of this stuff works at all is extremely impressive. No argument there. And, importantly, it being a currency, the central concept being money, is critical to the solution. You can't do it *without* it being a currency, because being money is what solves the problem.

Financial engineering

This is the single smartest thing anybody who is pro-crypto has ever said to me:

Saying that crypto is for money lovers is true, but just as trains are for transportation lovers and jackets are for clothing lovers, I see nothing wrong, it’s just a piece of human culture and interaction mechanics.

— JMD (@jmdiegog) January 3, 2022

The thing about having a successful model for creating a self-sustaining network of computers that model a currency is that for some people, that's enough. Some people find money and how it works intrinsically interesting. I spend a great deal of time building model cities and fighting fake wars in simulation games on my computer, so even though I find money boring I can imagine people finding SimMoney interesting. You can play with the parameters and experiment with new rules and game out the effect, which brings us to the next thing.

Entertainment

Crypto creates a massively multiplayer online game where the game is "currency speculation", and it's very realistic because it really is money, at least if enough people get involved. That means everyone's behavior is very realistic, and also if you win the game you have actual money that you can spend on other things you enjoy, like a 133 million dollar house.

NFTs add another layer to the game. Instead of just currency speculation, you're now simulating art speculation too! The fact that you don't actually own the art and the fact that the art is randomly generated cartoon images of monkeys is entirely beside the point: the point is the speculation, and winning the game by making money. This is, again, a lot of fun to some people, and in addition to the piles of money they also in some very limited sense own a picture of a cartoon monkey that some people recognize as being very expensive, so they can brag without having to actually post screenshots of their bank balance, which nobody believed anyway.

True cloud computing

Cloud computing providers like AWS and Azure are still (usually) tied to servers or specific instances: you are renting a certain amount of computing power for a certain amount of time. Those providers have terms and conditions and can cancel your contract. Running code on a cryptocurrency (called a distributed app or dApp) is different: once the app is deployed, it's possible for you to completely relinquish control of it, and it will run as long as anybody feeds it money. There's no way to take it down without taking down the entire network, which (per above) is highly distributed, so that's nearly impossible.

This is again a pretty interesting technical accomplishment; you can write an application that costs you literally $0 to run (because users pay in order to make it run).

Web3

Web3, as best as I can tell, is a vague term that encompasses using crypto technologies to create new versions of web technology. By virtue of being a currency, participants in such systems are rewarded instead of the inventors of the systems: a Twitter where your popular tweets earn you money, an Instagram where good photos directly support you. This is a lovely ideal but there are no extant implementations that approach this ideal, so it remains a cool thing to think about but I can't claim it as a good thing because it doesn't exist yet. There are lots of other definitions of web3, but none of them seem to exist either.

Overall

There is definitely something here. Huge numbers of people have been effectively incentivized to create a massive network of computers that can do arbitary tasks (I'm speaking of distributed apps, Bitcoin itself is much less interesting). As a technologist it's hard not to be impressed, and intrigued by the potential applications of such a system.

The bad

Environmental impact

The grievously wasteful nature of Proof of Work networks is the most serious objection to the most popular cryptocurrencies, including Ethereum (although it is trying to change) and Bitcoin itself. I also think it's the weakest objection, since there are extant examples of proof-of-stake cryptocurrencies that work. It seems if you want to do crypto things without burning the planet down, you can. Whether people will switch en masse to proof-of-stake chains depends on the answers to the other questions below.

Interactions at boundaries

This is in my opinion the biggest problem with a lot of the potential applications for crypto technology: the things that happen inside the network happen only inside the network. The interfaces to other systems, and to the physical world, are weak or non-existent.

A great example is NFTs: once an NFT is minted, the network contains a perfect record of who owns it and all the times it has changed hands. However, an NFT does not, cannot prove that the person who minted the NFT owned it in the first place. This is not a theoretical issue, with artists complaining that their work is being sold as NFTs without their permission. Even the cartoon apes were not sold by the people who made the art; the actual artists earned a flat one-time fee.

This presents a problem for the utility of NFTs, since it now requires that you either trust or verify that the original minter of the NFT owned the asset. And if you have to do that work anyway, or blindly trust, what is the utility of the NFT?

DAOs have a major boundary problem

DAOs also have this problem: if the purpose of the DAO is to have some real-world impact, such as buy a copy of the constitution or run an online community or whatever the fuck Friends With Benefits does (throw parties I think?), somebody has to actually do those things. Somebody has to show up to the auction and bid, somebody has to moderate the Discord, somebody has to buy chips and dips for the party.

The problem is that a DAO is not an employer or a legally binding contract. The DAO voting to do things has no legal weight. Even if you create a legal corporation to do the bidding of the DAO this doesn't get you out of the problem, because by law the people ultimately in charge must be a named set of human beings. Making it possible for software to employ humans as an independent legal entity is another neat idea but one that definitely does not exist right now.

So DAOs are ultimately reliant on humans who are gamely agreeing to do what the DAO says but aren't obligated to do so. This is strictly worse than starting a company or a non-profit to do whatever it is you want a DAO to do. It's more complicated and it's legally unclear at best.

Governance

Which brings us neatly to the next problem which is governance. All of the projects govern themselves differently but a common thread to many of them is that the size of investment (remember, everything in crypto is money) is proportional to your voting rights. In Bitcoin this investment has to take the form of actually buying computer time and hooking them up to the network; in others (and especially DAOs) there are actual voting systems.

This all sounds pretty democratic but in practice the majorities of tokens are always held by early adopters. Nearly a third of all bitcoins are held by 0.01% of its users (though Bitcoin doesn't vote by ownership, who owns the computers is much harder to figure out). The ENS early contributors control roughly 50% of the voting rights.

There's nothing intrinsically wrong with founders of projects controlling them, but it's not democracy, and it's identical to how startups work now. If anything, the concentration of power is greater and longer-lasting.

As with NFTs, the result is a much more complicated (and expensive) duplication of the existing system, so the utility of starting a DAO to perform an action versus starting a regular corporation or non-profit is unclear.

Unlike the fundamental boundary problems of NFTs and DAOs, the general governance problem seems solvable: you could establish a fairer system to distribute voting rights long term. But even if you have a totally fair governance system you still have the boundary problems.

Transaction costs and abuse at scale

Another fundamental problem with using crypto technology to build functioning software at scale is that the same good features that help solve the tricky problems begin to become anti-features at scale.

A big one is transaction costs. Imagine a Twitter clone that ran on crypto technologies. Everything that happens is a financial transaction. Presumably it would cost some very small amount of money to read tweets, maybe it could be free (it's very hard to imagine people paying just to read tweets), and it would cost money to post tweets. You can imagine some kind of system where the money earned from posting gets re-distributed to people whose tweets are read a lot, to encourage them to tweet more.

The problem is: how much should it cost? If it's very cheap to tweet, then the system will become over-run with bots, spam and other kinds of abuse. Twitter has huge and expensive systems to deal with all of this, CrypoTwitter has to figure out how to solve it with financial engineering. If instead you make it expensive to tweet, nobody will tweet, earnings from popular tweets will fall, those people will tweet less, and the network will die. (The same mechanic applies if you try to make the transaction following, or liking, or replying)

This goes for anything that wants to operate at scale. If you want billions of users to post millions of times per second, the transaction costs are going to have to be tiny. If the transaction costs are tiny, it becomes too easy to abuse. The bigger the scale of the thing you want to do, the harder this becomes.

Incentives for participation

Thinking about a theoretical CryptoTwitter brings up another big problem, again caused by the fact that the technology is also currency: incentives. The thing that keeps the computers in the network and keeps them cooperating is money. Either users must pay per transaction, and reward the computers that way, or there are tokens produced that have some intrinsic value and some kind of mining mechanism produces them.

It's hard to imagine how this would work for web apps as we understand them now. Consumers have come to expect that the cost of using a web app is free, the cost of reading news is free, the cost of getting email is free, sending messages to each other is free. Paying per transaction for these things creates a barrier to adoption.

Another way to do it would be to say that reading tweets and posting tweets are both free, and that tweeting itself creates tokens, maybe the tweets themselves are tokens. Then you can sell these tokens on a market. Why would anyone buy them? Not clear. Also, by making tweeting free, you've re-opened the hole for abuse mentioned earlier.

Maybe the money comes from somewhere else -- maybe subscriptions, or even advertising. But this brings us back to the same form of the previous problems: if CryptoTwitter is ad-supported, why run it as a currency? If users have to pay to use it, why would they do that when the former, ad-supported versions are free to use? What problem is solved by modeling a social network as a currency? What becomes easier?

Fundamentally: where does the money come from when the existing alternatives are free?

Web3 again

As Chris Dixon says in "Why Web3 matters", "Tokens align network participants to work together toward a common goal — the growth of the network and the appreciation of the token." That's great if what you're trying to do is create an ecosystem of tokens that appreciate in value all the time. What percentage of systems can be modeled that way?

The assumption of web3 seems to be "all of them", but it's not at all clear to me why that would be true. Why does playing a game, or making music, or watching a movie, or sending a message, or any of the millions of other things we spend time doing on the web make more sense or improve if modeled as a currency?

Lightspeed ventures has a roundup of the "web3/crypto/metaverse ecosystem" (when did the metaverse become the same as web3?) and every single project listed is a financial one: they are about getting paid, or investing, or selling.

Which isn't to say you can't marry the two: Axie Infinity is a sort of Pokemon clone where you have to buy your Pokemon, and you can earn real money by battling them and winning. Is this more fun than normal Pokemon? Maybe. But can every game be modeled this way? Certainly no single-player game could be; if you're the only person in the game, who can you transact with?

Fundamentally: is a currency a practical model for general computing?

The ugly

Finally there's the ugly, and there's no shortage of ugly. There are rug pulls and worthless NFTs and market manipulation of all kinds, such as wash trading. There are ICOs that turned into ponzi schemes, endless pyramid schemes, and of course good old fashioned theft. After all, it's all money.

I could go on and on about the huge numbers of grifters and scams but the important thing here is that the ugly and the bad are separate. Even if all the criminal elements and scamming were taken away, there are still the huge problems of boundary interactions, abuse, and incentives.

Conclusions

Maybe it's too early

Having lived through the web 1.0 and web 2.0 booms, one thing I'm very cognizant of is that the new thing does not rebuild the old things. The old things are still there; the new things are different. But at the start, people try to rebuild the old things. People tried to build "online magazines" before they figured out what web native publishing looked like. They tried to build "online shopping" with aisles and carts you pushed around. We built "bulletin boards" instead of forums.

It took a while for people to work out what the strengths and weaknesses of the medium were, and build things that went with the grain of the medium, that felt natural to users of that medium. Maybe that's why none of this stuff feels right yet; we are trying to rebuild the current web, trying to replace it instead of trying to augment it with new, different things (specifically, things that can be modeled as currency).

Maybe it's only finance

The less optimistic possibility is that financial engineering is all crypto can do. It may make money transfers more efficient, maybe it makes collecting royalties quicker and more transparent, maybe it's a great way to pool resources to invest as a group, maybe it's a great way to gamble with real money, either on fake monster battles or fake art. Those are all great, valid use cases. But they are narrow in scope: they were already financial transactions.

Maybe the only things you can model as a currency are financial things. Maybe crypto is fundamentally only useful for financial engineering. That's fine! Finance is a huge chunk of human life and it's full of inefficiency and greed right now. Making that simpler and quicker and fairer is a noble goal, and would be a great outcome. It really would change the world for the better, but in a much smaller way than the current hype would suggest.

There is, like I said, something here. But maybe not as much as people are hoping. Maybe it all comes back to this:

Saying that crypto is for money lovers is true, but just as trains are for transportation lovers and jackets are for clothing lovers, I see nothing wrong, it’s just a piece of human culture and interaction mechanics.

— JMD (@jmdiegog) January 3, 2022

Maybe crypto is only for people who like playing with money, its mechanics and rules. That's not all there is to the software or to the web. And that's not for me.]]> https://seldo.com/posts/what-i-ve-learned-about-data-recently com.seldo/posts/what-i-ve-learned-about-data-recently Mon, 21 Jun 2021 22:59:41 GMT I've been a web developer for 25 years now. In the last 10 years, at two different companies, my focus has increasingly shifted to working with what is somewhat eye-rollingly referred to as "big data" but I will just be calling "data". In the last 18 months, since I joined Netlify, I feel like I have really leveled up from "just do whatever works" to feeling like there's a Right Way to do it.

I like to share when I learn things, so that's what this blog post is about. But admitting that I've learned things involves admitting not knowing things. Admitting I spent 8 years doing data things without using what in retrospect feels like basic tooling involves a certain amount of vulnerability. But whatever, I've done dumber stuff and more publicly. If this post teaches you nothing, well done. If you learn something, I hope this helps you climb the learning curve faster than I did.

Learning 1: you want engineers, not scientists

I learned this one before I was even hired at Netlify. While deciding about my next career step, I interviewed at a bunch of companies who were doing data-related things. Some were very sophisticated in how they handled data, but they were a minority. There was one pattern in particular I saw repeated over and over: a fast growing company would say "wow, we have all this data, we should do something with it". Then they would go out and hire a Data Scientist, ideally with a PhD, as their very first data hire. This is how that is imagined to work:

This is a mistake. Firstly, the term "Data Scientist" is over-used to the point of being meaningless. To me, a data scientist is an academic with very advanced degrees. They advance the state of the art in modeling, they solve novel problems in machine learning. They write white papers in journals. This is not what you need. Unless "we solve a new problem in machine learning" is what your company was founded to do, this is not what you need done. What you need is a data engineer.

The problem you actually have is: "we are generating a ton of data, but we have no idea what's going on". On the face of it, it sounds like somebody called a "data scientist" is the right person to solve that! But to actually solve your problem, what you need to do is:

1. Work with the engineers on your team to instrument your code so it's generating usable data in the first place.
2. Extract all your data from the random places it's accumulating, in a zillion incompatible formats
3. Load it all into some kind of central store, so that you can compare and correlate data coming from different sources
4. Transform the data from its raw state into answers to questions, either as a one-time analysis or as an ongoing report or dashboard
5. Figure out what the right questions are for the business to be asking, and what strategic changes are dictated by the answers you get.

(Side note: there is a small revolution ongoing in the data space about the shift from "Extract-Transform-Load" to "Extract-Load-Transform"; the difference is that previously your central store was probably not big enough to hold all of your raw data, so you'd need to aggregate it down first, i.e. transform it, before you could put it into your central store, now your central store is so huge you can just throw everything in there as-is, and transform it later, i.e. load it first. It is quite a lot of fuss for such a simple change.)

Where the person I call a "data scientist" comes in is step 4, and not most of the time. Maybe 5% of the time the kind of advanced analysis a data scientist can bring to bear is going to provide utility over what a simpler, more straightforward analysis could provide. The problem is, if your first hire is an academic with a PhD, they will never even get to step 4. They will get stuck in steps 1-3, getting increasingly frustrated, because those things are not their primary skill set.

What you need instead is engineers and analysts to take care of each step. In practice, you might not have a separate person for each of these roles, and even if you have a larger team, people will skip back and forth between the somewhat arbitrary divisions I've created here. In the real world, your analysts have to do engineering sometimes, and your engineers will be more effective if they can predict how the analysis is going to go. It's going to look more like this:

Learning 2: running a data warehouse is not your job

Another pitfall tech companies run into, when they have a data problem, is thinking because they are a tech company, and running a data warehouse is technical, that's a thing they should be able to do. Again, like hiring a data scientist for data, it seems logical! But data warehouses are their own complicated beasts. They do not follow the same operational constraints and they have different scaling properties to whatever databases it is you're running in production.

Unless what your company does is run data warehouses, you should be buying your data warehouse from somebody else. Any money you save from not paying for a data warehouse service will be eaten by the engineering time you spend operationalizing a data warehouse, and it will never be as fast and scalable as one run by experts. I don't have strong opinions about which data warehouse to use -- I've used Redshift, AWS Athena, Databricks and Snowflake -- but I do believe strongly that you should get someone else to run it unless you are a very big company.

Learning 3: there are frameworks now

For the first 10 years of messing with data, my "data pipeline" code was always a roll-your-own deal. Data would be accumulating in logs somewhere, it might be in CSV format, or Apache log format, or maybe newline-delimited JSON. It might be one file per day, or per hour, or per minute. The files might be any kind of size from a few megabytes to thousands of gigs. I would write some code in whatever language happened to be lying around to parse and extract this, maybe get some cron jobs going, load it into a database that was lying around handy, maybe slap some graphs on top. It worked! But it was its own special snowflake, and it seldom scaled.

We had the same problem in web development -- you could build a website an infinite number of ways, and that was the problem, because no two people would pick the same way, so they'd fight about it, or misunderstand how it worked, and everyone was constantly rewriting from scratch because they couldn't figure out the last thing.

This changed -- at least for me -- around 2006, when I heard about Ruby on Rails. Ruby is not my favorite language, but Rails was a revolution. Rails provided an opinion about the Right Way To Do It. It wasn't necessarily always really the best way to do it, but the surprising lesson of Rails is everybody doing it the same way is, in aggregate, more valuable than everybody trying to pick the very best way.

The revolution Rails and the blizzard of frameworks that followed it enacted was that you could hire somebody into your company who already knew how your website worked. That was a gigantic breakthrough! Instead of spending half your time fighting bugs in your architecture, the framework maintainers mostly took care of the architecture and you could fight bugs in your business logic instead. The result was a massive reduction in work for any individual developer and a huge acceleration in the pace of web development as a whole.

Rails was invented in 2004, but as I mentioned I came across it in 2006 and felt somewhat like I was late to the party. There was a Right Way To Do It now, and I'd spent 2 years doing it the "wrong" way! Damn! Better catch up! I never actually ended up building a site in Rails (I simply did not enjoy Ruby), but I picked up copycat frameworks in other languages and was off to the races.

Coming across the two frameworks for data engineering in 2020, Airflow and dbt, felt very much like finding Rails in 2006, except I'm even later to the party. Airflow and dbt were both first released in 2016.

Airflow

I'm not going to talk too much about Airflow simply because I don't know that much about it. In the diagram above, I live in steps 3-5, and not even 3 that often as our team has grown. Airflow lives at steps 1 and 2. It allows you to create chains of data extraction jobs that depend on each other, to pull piles of random files continuously from all the places they can go, transform them into tables, and put them where they need to be. It handles all sorts of niceties like timing, error detection, retrying, and more. Very importantly, the configuration of Airflow lives mostly in Git, meaning you can have version control, code reviews, and all the things you'd expect to have of any other critical part of your infrastructure, and which have been notably absent from many of the home-grown solutions I've seen.

It's all a huge leap forward from cron jobs and bash scripts, and the members of my team who take care of Airflow for us are heroes. The tables created in Airflow are the foundation on which dbt rests.

dbt

dbt, which stands for "data build tool", is the framework I know more about. At its most basic, dbt allows you to write SQL queries which automatically become queryable views. Those views can be referenced in other views, and those further referenced, allowing you to build an elegant tree of data provenance from its original raw form to collated, aggregated, filtered collections that answer your questions. If you make a change to an upstream view, the downstream views are automatically refreshed.

On top of that, you can selectively declare a view to be "materialized", i.e. made into a table rather than a view. You might want to do this for performance reasons, since views that depend on views can get quite slow. Every time new data arrives in the upstream views, the downstream tables will be automatically rebuilt by dbt, so you never end up with stale data. If that rebuild process begins to get slow, you can further declare a table to be "incremental", meaning only new data will inserted or updated, rather than replacing the entire table each build.

Finally, you can declare basic tests for all your tables -- is this column supposed to be unique? Are nulls a problem here? You don't need to write any test code, you can just declare these features as a property of your view or table and every time your tree of tables is rebuilt, those properties will be checked.

And like Airflow, all of this configuration and declaration can live in Git, so your team can have version control, code reviews, staging builds and everything else a modern engineering team expects when collaborating.

None of this is ground breaking. Anybody who's built a data pipeline has come up with some way to declare that tables depend on other tables, to rebuild them, and to test them. What dbt does, like Airflow, is a provide a known way to do that. You can hire people who know dbt and Airflow and they can be productive on their first day, without you having to guide them through your special snowflake of a data architecture. It's simultaneously simple and yet as revolutionary for the data team as Rails was for the web team.

Learning 4: a data team is nothing without domain experts

My final learning is cheating a little bit, because I knew it already, but I have seen a great number of startups hiring for data fail to understand it: your data team is good at data. They have a great number of specialized skills for wrangling it, querying it, making sense of it, and visualizing it. They are experts at data. What they are not experts at it your business.

The data scientist hired in the original diagram will flail at steps 1-3 but even when they get through them they will flail again at step 5. You know what your goals are; you know who your customers are; you know what they are trying to achieve; you know what your product is capable of, and what it does not do. Your data experts do not know these things, at least at first. A data team hired in isolation without working closely with domain experts from within the business is going to do a huge amount of work and produce nonsensical results based on inaccurate assumptions. A data team without close partnership to ask the right questions and feed back on results is useless. This is why I went to work as a data analyst at a web company; I know the domain, so I can frequently ask the right questions.

Still learning

While all of this has been great fun to learn and apply, and I feel a hundred times more productive as a data person than I did 18 months ago, there are still some big, unsolved problems that I don't have answers for yet:

1. Should the business self-serve data?

Data tool vendors love to sell that their fool-proof SQL interface means that stakeholders can "write their own queries" or even "build their own dashboards with no code". I'm split on the question. Lots of people know SQL and can make useful progress on their own, and data teams shouldn't gatekeep them. On the other hand, querying the wrong table, or misunderstanding a result, can produce inaccurate conclusions that a dedicated data person wouldn't make. So is self-serve worth it? I don't know. My current compromise is that I recommend people try to self-serve and then run what they've built past the data team to check their work, but it's not a perfect solution.

2. What about dashboards?

Everybody who does an analysis once ends up wanting it run all the time. The solution is dashboards: you create a query that lives at a URL, and it can be revisited and refreshed at will. In practice, dashboards get stale very quickly as the business changes around them, and are seldom documented well enough for it to be obvious when any particular dashboard should no longer be considered reliable. Also, dashboards accumulate over time, making it difficult for newcomers to find the most important data. Dashboards are great but maintaining dashboards seems to be a full-time job, and the information about which dashboards are important and reliable lives in the heads of the data team, which is not a good solution.

3. How should discovery work?

A business of any size breeds data at a bewildering rate. New products, new features, new pricing, new analyses can quickly produce hundreds of tables. A new data team member, or even an existing one, can find it impossible to know which tables are suitable for which purpose, and which data lives where. dbt addresses this with documentation of tables in the code, down to the column label, and auto-generating searchable documentation. But being able to find every column that contains the word "revenue" and knowing which table is the right one to answer the specific question about revenue you have today are very different things. Again, the answers to this currently lie in the collective knowledge of the data team, and that's not a scalable answer.

We are of course not sitting on our hands perplexed by these problems, and are trying many solutions, including process changes and third-party tools from the exponentially-growing collection of data tool vendors who email us every day. But I don't like any of the answers I've seen yet.

More to come

The last 18 months have been tremendous at up-leveling my understanding of how a data team should be built and run and I'd like to call out Emilie, the manager of Netlify's data team, who joined six months after I did and has been responsible for introducing me to a great deal of this stuff, as well as hiring a great team to get it all done.

As I was with web frameworks, I feel like I have arrived late to the data management party, but it's a party nevertheless. I look forward to another year of learning. ]]>