Why do some things work, while others don’t? Why has SpaceX succeeded with more than 99 percent of its Falcon 9 launches while embarrassing software failures like Healthcare.gov in the public sector remain all too common? Two books I’ve just read go a long way toward an explanation.

I didn’t expect the books to have anything in common. The first was Walter Isaacson’s biography of Elon Musk, which I picked it up for recreational reading after enjoying his books on Jennifer Doudna, Leonardo da Vinci, and Benjamin Franklin. The second was Jennifer Pahlka’s Recoding America: Why Government Is Failing in the Digital Age and How We Can Do Better. Excellent though it is, Recoding … is nobody’s idea of beach reading.

Waterfalls Without Water

Pahlka’s story begins with a waterfall. Not the sparkling fall of a pristine mountain stream. I imagine this one more as a stairstep of concrete dams and pools in some aging industrial setting. 

In reality, there is no water at all in this particular waterfall. Instead, it’s a long-established model of project management that gets the name from its strictly one way, top-to-bottom operation. The waterfall breaks down a project into discrete steps — setting requirements, design, implementation, testing, deployment and maintenance. Those working at each stage take the decisions that come down from the earlier stages as givens and carry on.

The waterfall works best where requirements are clearly defined and midstream changes aren’t needed. When I asked ChatGPT for a successful example, it suggested NASA’s Ingenuity helicopter. I don’t know the details of that project, but it certainly fits in one respect: if the machine hadn’t worked on the first try on Mars, there is no way anyone was going to make a house call to fix it.

Unfortunately, as Pahlka details with case after case, the waterfall doesn’t work well for government software development. The requirements are set by Congress or an agency of the executive branch, or first set by the one and refined by the other, before being passed on to others for design and implementation. The problem is that in carrying out their steps, actors at each stage are likely to discover that some of the requirements cannot be met, or can be met only at excessive cost — or are poorly aligned with the needs of the ultimate users.

When that happens, it can be a slog to go back to the responsible agency, let alone to Congress, to get the requirements changed. At best, the project is successfully completed with cost overruns. At worst, it doesn’t work at all, or proves useless to its intended users. As one critic quoted by Pahlka puts it, “The waterfall method amounts to a pledge by all parties not to learn anything while doing the actual work.”

The Way of Musk

Elon Musk’s approach to project management could not be more different. By Isaacson’s account, “requirements” is a dirty word in Musk-world. When something isn’t going right, and one of his engineers cites a “requirement” as the reason, Musk goes ballistic. Who created that requirement? Answering “the legal department” is not enough — he wants an actual name. Why do we have to do that requirement? At SpaceX, the only requirements that can’t be questioned are the laws of physics.

The need to question requirements has several corollaries that fit together into what Isaacson calls Musk’s “algorithm.” One corollary is that those who work at higher levels of the organization have to be in close touch with those working below. Here is Musk, at 2:30 a.m., lying underneath a car moving along the Tesla assembly line. “Why do we have four bolts here? Who set that specification? Can we do it with two? Try it!”

Another corollary is to test things quickly and learn by failing. Tom Mueller, the chief engine designer at SpaceX, grew up playing with toy rockets in Idaho. The rockets he designs now are not toys, but he’s still not afraid of moving fast, blowing up a few and trying again. “It’s not how well you avoid problems,” he says. “It’s how fast you figure out what the problem is and fix it.” The Falcon 9, now famous for its 99 percent success rate, blew up the first three times it was launched.

In the world of software, the Musk-like approach is enshrined in the Manifesto for Agile Software Development. The agile model turns the waterfall upside down. Instead of beginning with requirements set by those at the top, the first of the manifesto’s 12 principles is, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Its second principle is to “Welcome changing requirements, even late in development.” The collective encourages the use of small teams where people with different specialties can talk face-to-face and work fast. Whenever you encounter a simple, user-friendly website that gives you overnight delivery of a bag of dog food, chances are agile was behind it.

Unfortunately, it is not easy make the agile model work in government. Much of Pahlka’s book is spent explaining why. We have already encountered the difficulty of changing requirements, once they are set by Congress or executive agencies. A pervasive procedure fetish another big problem: bureaucrats are not rewarded for their agility in satisfying their customers, but for following the rules. If a product fails, their jobs are safe as long as they can prove that their decisions, even the disastrous ones, followed the straight and narrow. Still another problem is a legalistic mindset that turns every decision into a jury trial. That may be appropriate for high-level decisions, but it destroys agility when it bleeds into the details of implementation.

Privatization: How Matters

So, if the government is hopelessly stuck in waterfall mode while the agility model thrives in the private sector, isn’t privatization the answer? One problem: the development of government software is already privatized. Pahlka traces privatization all the way back to OMB Circular A-76, issued in 1966, when “data processing services” mainly meant things like keypunching numbers onto paper cards. Since then, she says, “government has thought of digital technology much like the pens and paper clips that GSA buys for government offices — something government would be crazy to produce for itself.”

Privatization is great, provided you privatize the right things in the right way. Sure, NASA did the right thing in contracting with SpaceX to deliver cargo and crew to its space station, rather than developing its own replacement for the Space Shuttle. But the way data processing services have been privatized has not worked nearly as well.

The prevalent approach to privatization breaks the waterfall at its first link, with Congress and agencies setting requirements on the upstream side and private contractors carrying out design, implementation and the rest downstream. I like to think of the break between the public and private roles as something like the blood-brain barrierin the human body. Maintaining appropriate separation between the brain and the bloodstream is essential for health. But sometimes the barrier works too well, hampering efforts to deliver medicine where it is needed.

A number of factors cause the barrier between government policymakers and private contractors to work too well. To begin with, the original contracts between government agencies and private service providers have to go through a time-consuming procurement process that Pahlka characterizes as “more complex and technical than the latest programming languages.” The result: carefully crafted requirements are often obsolete before the contract is even signed.

Later, when contractors request changes, agencies cannot deliver them without going through public notice and comment procedures that can eat up months or even years. Getting changes through Congress can take even longer. The whole system stalls.

The solution, says Pahlka, is not to give up on privatization, but to build the digital competencies on the government’s side of the barrier that it needs to achieve at least a modicum of agility. In part, that can be done by changing policies. She names several successful examples, such as increasing the powers of chief information officers within agencies, and opening alternative, less burdensome procurement pathways. But primarily, she says, building digital competencies is a matter of investing in people.

Her book is full of individual heroes who have staved off disaster within the badly flawed system. On an institutional level, she points to the establishment of the U.S. Digital Service under the Obama administration. USDS experts collaborate with public servants throughout the government to share best practices.

A further people-oriented investment has been to bring product managers into government to work side-by-side with the project managers who have dominated digital procurement in the past. Project managers focus narrowly on the “how” of coordinating and overseeing the flow of work, something that is, indeed, important. Product managers have a different role that focuses on “what” and “why,” especially on whether the product under development truly serves customer needs.

• • •

In truth, the government is never going to be as agile as the private sector. And up to a point, that is good. We would not want to let an incoming president fire 90 percent of government employees overnight, as Elon Musk did when he took over Twitter. But neither do we want the government to be so rigid that it fails wretchedly to satisfy us, its citizen-customers. Pahlka tells a horror story of a Department of Veterans Affairs healthcare website that worked perfectly and met all its requirements — as long as it was accessed using the specific software and hardware configuration installed at VA headquarters. No one noticed (or seemingly cared) that ordinary vets with critical health needs could not reach the site from the outside.

Pahlka thinks we can put disasters like that behind us. She points to the stunning simplicity of www.covidtests.gov as an example of what can be done with a little competence and creative thinking. Let’s hope for more of that.