Download: PDF | EPUB

Tech development is often portrayed as an open road — a smooth freeway of continuous progress. But the truth is that the road of technology is more like a suburban street, paved with numerous unmarked dead ends.

To see one of these dead end, look at roads themselves. As a method for moving people, roads work well … until you fill them with millions of cars. Then, even the most massive freeway becomes a cul-de-sac — a dead-end technology out of which we must reverse.

This dead-end problem happens everywhere, and follows a predictable narrative. Faced with an immediate problem, someone hacks together a solution. Other people then copy the hack, causing it to slowly get institutionalized. Later, we realize that the hack created a host of downstream problems, and so we look for solutions. But by then, the hack is ‘locked in’. So instead of rethinking the original hack, people solve their problems by piling hacks on top of hacks. A cascade of hack hacking follows.

Along the way, a few people realize that the problem is the original hack. And so they look for a new route that avoids the hack cascade. “Hmm … adding another freeway lane never solves our traffic problems, maybe we should try building something different? … Something like public transportation?”

“Nonsense,” the hack hackers say. And they continue hacking.

Eventually, the hack cascade becomes so comedic that almost everyone realizes that the problem is the original hack. But during the interim, there’s much confusion.

Turning to computers, it seems that software development is in the midst of a hack cascade.

The roots of the problem were sown in the 1970s, when programmers at Bell Labs created an operating system called ‘Unix’. Although the OS had many great features, it also had some elements that lacked foresight. In particular, Unix developers dumped all of their programs in a single, flat directory.

Had this hack been temporary, it would have been no big deal. But it was not temporary. Instead, the hack got institutionalized. Then, around this institutionalized hack, computer software grew much more complicated. And so a hack cascade followed.

Today, system administrators manage computers that contain thousands of programs – each one with an interlocking set of ‘dependencies’ on other programs. And admins wrestle with this task using a design that was never intended for a job so complex. Of course, there are tools that make software management easier. But most of these tools are like new lanes on a freeway — they layer hacks upon hacks, without rethinking the original problem.

Looking at the situation (in 2006), a Dutch computer scientist named Eelco Dolstra argued that there was an obvious solution: nix the Unix design and replace it with a warehouse-grade approach to managing software.

Dolstra’s core idea was elegantly simple: instead of dumping programs in an unorganized flat directory, we should file them according to a unique ID. Once we have this ID, our unruly pile of programs gets transformed into a meticulously organized warehouse, managed via a central database. And from there, a host of software-management problems simply vanished. Magic.

Dolstra called his design ‘Nix’, and joked that it would ‘surely conquer the world’. I think he was right. Just as today it’s unthinkable to manage a large business without a database for tracking inventory, it will one day be unthinkable to manage software without a Nix-like, database-driven warehouse.

Your computer has wheels

To dive into Nix and the world of software management, let’s start with something more accessible: the wheels on a car.

When you drive a car, you take it for granted that it has a set of wheels which allow it to operate. But did you ever stop to think about how those wheels got there? Maybe not. But somebody did. At some point, someone ordered the proper set of rims and tires and installed them on the car.

In software parlance, we call this work ‘dependency management’. Since cars depend on wheels to operate, someone has to ensure that the correct wheels get matched to the correct car.

Zooming out to the big picture, ‘dependencies’ happen not just with cars, but with all technologies. And that’s because technology is iterative — it builds on past solutions. Since the wheel already exists, there’s no need to reinvent it. So if you want to move a car across land, you take a set of already-invented wheels and bolt them in place.

Similarly, when you write new software, you don’t solve every problem with a fresh batch of code. That’s because like wheels, many basic computational tools already exist. So instead of reinventing these ‘wheels’, you simply bolt them onto your code.

Here’s a bolting-on example. In modern computing, a popular task is to render 3D graphics. (That’s how we get pretty video games and nice movie effects.) As you’d expect, there’s lots of new software devoted to this task. Think of this software as a sexy sports car. On the surface, it’s shiny and new. But under the frame, it’s supported by a much older technology. In this case, the ‘wheels’ of our rendering engine are software libraries that crunch linear algebra.

Backing up a bit, the task of rendering 3D graphics seems, well, ‘graphical’. But really, it’s a math problem involving lots of matrix transformations — a task that mathematicians call ‘linear algebra’. Now, the thing about linear algebra is that it’s also a basic part of physics. So when early computers were put to use solving physics problems, early programmers wrote lots of code to do linear algebra.

In the late 1970s, this code was formalized in a set of Fortran libraries called BLAS — short for ‘basic linear algebra subprograms’. Today, these BLAS libraries are a ‘wheel’ of modern computing. So when your fancy new rendering engine needs to do linear algebra, you don’t write new code. Instead, you call a BLAS library to do the job.¹

By invoking BLAS, you save yourself a lot of work. Still, the ride isn’t free. Once this ‘dependency’ has been evoked, you’re left with the task of managing it. So when you drive your rendering-engine ‘sports car’, someone has to make sure that under the frame, it comes equipped with the right BLAS ‘wheel’.

Software part numbers

To state the obvious, when you buy a sports car, you can’t fit it with tractor wheels. The car clearly needs a specific type of wheel, described by a specific part number.

Likewise, when you put dependency ‘wheels’ under your software, you require a specific set. It’s (largely) for this reason that software developers give their programs version numbers — essentially part numbers meant for other software mechanics.

Now, I’m not much of a mechanic, but I can tell you about some of the software parts that I’m using. I browse the internet with Firefox version 122.0.1, I do statistical analysis with R version 4.3.2, and I edit text files with Neovim version 0.9.5.

As a user, these part numbers are pretty boring. When I drive my car, I don’t care about the part number of the tires. Likewise, when I use Firefox, I don’t care about its version number. In both cases, I just want the thing to work. But for software developers, version numbers are essential. Just as your mechanic must know which tires to put on your car, software developers must know about the dependency versions that are compatible with their code.

Returning to the task of rendering 3D graphics, I might subcontract the job of matrix number crunching to a BLAS library — say OpenBLAS. But because OpenBLAS changes over time, I’ve got to note the particular version that’s compatible with my code.

Once that’s done, I have a maintenance job on our hands. Every time my rendering engine gets installed, I must ensure that the computer comes with the correct version of OpenBLAS. If this maintenance job fails, my rendering engine may implode.

Breaking changes: a Python tale

The word ‘may’ (implode) is important. Sometimes a dependency mismatch is no big deal. It all depends on the state of ‘backwards compatibility’.

Turning to the wider world of technology, I can easily fit a modern bow with an arrow from the Stone Age. In software parlance, we’d say that modern bows are ‘backwards compatible’ with ancient arrows.

Returning to software, the king of backwards compatibility is undoubtedly Windows. If you take a legacy app designed for Windows XP, it will likely run just fine on Windows 11. And that’s by design. Microsoft knows that its business clients run a host of legacy software. So it bends over backwards to make sure that new versions of Windows are backwards compatible with old programs.

Now in principle, it would be nice if all software was forever backwards compatible. But in practice, this compatibility limits innovation. For example, if you want to design a rifle, you’ll have to break backwards compatibility with arrows. That’s because the whole point of a rifle is to explosively shoot something hard and compact — something distinctly not-arrow-like.

Similarly, giving your software new features often means breaking backwards compatibility. Python is the poster child for this approach. In the late 2000s, Python developers decided that the language had some features that were sub-optimal. (Most notably, the print command was a ‘statement’, not a ‘function’.) And so they implemented a bold redesign called ‘Python 3’, which infamously broke backwards compatibility with ‘Python 2’.

Chaos ensued.

For developers, the change meant that they had to refactor their Python codebase to meet the new standard. But since this refactoring took time (sometimes more than a decade), it left software maintainers with a hodgepodge of different Python code.

Suddenly you couldn’t just tell the computer to execute a script with ‘python’. Some code demanded ‘python2.7’, while other code required ‘python3’. It was … a mess.

A shebang non-solution

To quote Richard Feynman, ‘surely you’re joking’. On its face, it seems trivial to keep track of two versions of the same software. Can’t we just create some sort of symbol that tells the computer which version of Python to use?

Actually, there’s a tool for this very task. It’s called a ‘shebang’, and its sole purpose is to tell the computer which program to use when executing a script.

For example, if I wrote a script that should run on Python version 2.7, I’d put the following ‘shebang’ on the first line:

#!/usr/bin/python2.7

Likewise, if my script needed Python version 3, I’d use this shebang:

#!/usr/bin/python3

So with our shebang, it seems like we’ve got an easy solution to the dependency problem. We simply tell the computer about the Python version required for each script, and the computer sorts things out. Problem solve, right?

Oddly, no.

The issue with shebangs is that they are largely shouting into the void. Sure, they can tell your computer that Script A requires Python 2.7, and Script B requires Python 3.0. But here’s the problem: by design, your computer has only one of these Python versions installed. So any script that doesn’t call the installed version of Python will simply fail, shebang be damned.²

Unix hieroglyphics

To understand why shebangs are largely shouting into the dependency void, it helps to take a closer look at their form.

Returning to our Python shebang (#!/usr/bin/python3), the text ‘/usr/bin’ is a hieroglyphic-like reference to a half-century-old OS design. Although the code seems opaque, it’s actually quite simple. It’s the equivalent of a folder called ‘my programs’.

The etymology of ‘/usr/bin’ traces to the developers of Unix. Working in the 1970s on terminal-only computers, Unix designers had a preference for terse names. So when they needed a folder to house executable binaries, they called it ‘bin’. And if these binaries were for the user, then the ‘bin’ folder got put in a super-folder called ‘usr’. Hence, we get ‘/usr/bin’ — the folder housing the user’s binaries.

Now, due to quirks of history, this folder structure got baked into most of the world’s computers. So today, when you tell your computer to run Python, the shebang says to look in the directory ‘/usr/bin’.

So that’s the hieroglyphics. Now here’s the problem they reveal. When we actually look inside the folder ‘/usr/bin’, we expect a warehouse-grade design — one that can handle thousands of programs, each with dozens of interlocking dependencies. But that’s not what we find. Instead, ‘/usr/bin/’ is more like a single desktop, piled high with sparsely labeled documents.

Historical hacks

Humans have a quaint tendency to view history with rose-colored glasses. Today, I’m flying by the seat of my pants. But in twenty years, I may recall the same decisions as being ‘well planned’.

When it comes to institutional decision making, this rose-colored vision is even more potent. Once decisions get institutionalized, they acquire an aura of sacredness, as though they follow deep principles. But usually, this aura is bullshit. Most decisions are sparsely-planned hacks designed to get the job done.

And that brings me back to Unix. Because it has influenced so much of modern computing, the Unix design has an aura of sacredness. But the truth is that Unix developers made plenty of hacks. And the programs directory ‘/usr/bin’ is one of them.

In hindsight, modern software demands a warehouse-grade system for organizing files. But the early Unix programmers weren’t thinking about the future of computing. They were just trying to hack together a system that worked.

“Hmm,” they said. “I have some binary programs. Where should I put them? How about I shove them in a folder called ‘bin’.”

Done. But then came more problems.

“Damn, the tape drive is full. I need a new place to put programs. Hmm … let’s make another folder called ‘sbin’. We’ll put ‘system binaries’ in that one, and store it on a different tape drive.”

Done. But still more problems.

“Damn, the new tape drive is full? No worries, let’s make another folder called /usr/bin. We’ll put user binaries there, and store it on yet another tape drive.”

And so on.³

Now these words may be apocryphal, but the gist is not. The truth is that Unix developers were hacking together an OS that worked on computers with extremely limited storage. They had no pretense that the design was sacred. Instead, they acted like clerks given a handful of documents. “Where should I put these files? Well, for now I’ll shove them on my desk.”

For a while, the system worked just fine. Then came the software deluge.

As you can imagine, a shove-papers-on-your-desk filing system doesn’t work well when you’ve got thousands of documents to manage. The same goes for software. Looking at the Unix method of shoving programs into a flat, unorganized directory, it’s comically ill-equipped to handle the needs of modern programs. And yet it is the defacto standard. Why? Because the Unix design got institutionalized.

Now, this institutionalization is its own story — one that I tell briefly in the appendix. The short version is that during the early 1990s, it looked like Unix would die along with mainframe computers. But then came the internet and smartphones, both of which are completely powered by Unix-like systems. The result is that today, the Unix design (warts and all) dominates modern computing.

To be fair, Unix has lots of fantastic features. (Pipes spring to mind.) But its system for organizing software is not one of them. This system was a hack, pure and simple. And once the hack got institutionalized, it led to a hack cascade.

The hack cascade

Having inherited a 1970s filesystem design, programmers in the 1990s were tasked with using it to manage an increasingly complex software landscape. It was like playing a game of catch with both hands tied behind your back.

The root of the issue is that different programs often have mutually inconsistent dependencies. (Script A requires Python 2, but Script B requires Python 3.) The two-handed approach to solving this problem would be to have multiple versions of said dependencies, each served to the correct program. But the inherited Unix design says no, that’s not an option. Instead, you need to tie back your hands and somehow make the system run with a single version of each program.

Surprisingly, developers managed to make great strides with this no-hands catch. The result is the standard Linux distribution.

The Linux distribution

You can think of Linux as an open-source re-implementation of Unix. The project got started in the 1980s (with GNU), but became supercharged in the 1990s by the internet. With Linux and high-speed internet, we had a free operating system that could run free software, distributed for free.

The catch was that managing this software was difficult. You had to track down dependencies, install them yourself, and ensure that you had no clashes. Because this task was tedious, Linux developers soon came up with an automated solution: the package manager. Instead of tracking down dependencies by hand, the package manager did the work for you. Pretty cool.

Still, there was the problem of technological lock in. Stuck with the legacy Unix design, distribution maintainers had to make difficult decisions on the back end. “Hmm … the newest version of Firefox depends on Widget X, version 1. But the newest version of LibreOffice needs Widget X, version 2. How do I manage this conflict?”

The community hit on two basic solutions. The first was to be conservative about updates. On the backend, distribution maintainers made sure that all of their software had consistent dependencies. The catch was that this matching typically required serving software that was out of date. The second approach was to throw caution to the wind and roll the most up-to-date software. The trade-off here was that sometimes these up-to-date programs would break due to dependency mismatch.

Today, this trade-off is alive and well. When choosing a Linux distribution, the standard thinking is that you can either have software that is stable but out of date, or you can have software that’s up-to-date but prone to breaking.

Software containers

In the late 2000s, as dependency management grew more complex, developers started to look for alternatives to the distribution model. How, they asked, could they keep the Unix design while adopting a better way of managing software? The hack they came up with was to put software into ‘containers’.

Here’s the general idea. Having locked in the Unix way of organizing files, programs have expectations about where their dependencies are located. For example, a program’s binary dependencies will be in the folder ‘/usr/bin’, its library dependencies will be in ‘/usr/lib’, and its shared files will be in ‘/usr/share’. Now in standard Unix systems, these directories are shared by all programs. But in the container approach, we give each app its own mini version of the whole filesystem.

The advantage of creating this container is that we can sidestep the shortcomings of the Unix design. In the system folder ‘/usr/bin’, we can house a single version of each program. But if we containerize the filesystem, then each app can have its own version of ‘/usr/bin’. And there, it can store any version of any dependency. So in the Firefox container, we can put Widget X version 2. And in the LibreOffice container, we can put Widget X version 3.

With this container design, we avoid the trade-off between stability and currency. Each container can run the most up-to-date version of its app without fear of breaking other programs. Another advantage is that containers make software portable. Since the app brings with it a whole mini filesystem (and dependency chain), the thing can run in almost any environment.

Because of these advantages, the 2010s saw a rush to containerize software. Today, most smartphone apps run in containers. And on the server side, container software like Docker and Kubernetes has become wildly popular.

Still, containers are not flawless. Their biggest drawback is that they create software bloat. Here’s why.

Suppose, that you have dozens of programs that all require the C library glibc. In a standard Linux distribution, these programs share a single copy of the library. But if you package these programs into dozens of containers, each container gets its own version of glibc. And so you end up with (potentially) dozens of unnecessary glibc copies.

Is this duplication a problem?

No and yes. It’s not a problem in the sense that today, storage is profligate and cheap, so duplicating dependencies isn’t game ending. But duplication is a problem in the sense of being an obvious design flaw. Can’t we have software that is stable, up-to-date, and efficient at sharing resources? With containers, the answer is no.

Another problem is that while containers work well for managing user-facing apps, they work poorly for managing the core components of an operating system. That’s because by design, these components need to function as an interlocking system, so putting them each into containers defeats their purpose.

Nixing Unix

To summarize the story so far, the Unix legacy has left us with a cascade of hacks, each one seeking to to ease the burden of managing software. Like adding lanes to a freeway, these hacks seem to work initially. But over time, they reveal more problems.

In hindsight, the obvious solution is to stop the hack cascade, and revisit the original hack that got us into this mess — the Unix decision to dump programs into an unorganized, flat directory. But the trouble is that once you’re enmeshed in a particular way of thinking, ‘obvious’ solutions are not at all obvious. And that’s why Silicon Valley developers rushed into a cascade of hacks. Few (if any) of them considered the ‘obvious’ solution of nixing the Unix design and rebuilding something better.

That job was left to a Dutch PhD student named Eelco Dolstra.

Working out of the University of Utrecht, Dosltra wrote a 2006 thesis that (like most academic texts) was read by almost no one. But in hindsight, Dolstra’s dissertation contains a brilliant solution to the problem of managing software.

The gist of his argument is simple: if we’re managing lots of programs, we should have an organizational system that’s up to the task. Instead, of stuffing programs into an unorganized flat directory, we should create a numbered filing system for organizing and tracking software. And the way to create this system, Dolstra argued, was to use ‘cryptographic hashes’.

Software, hashed

Although cryptocurrency has given the word ‘crypto’ a bad name, cryptography has many useful applications — including (it turns out) managing software.

To dive into Dolstra’s idea for using cryptographic hashes to organize computer programs, let’s start with the concept of hashes themselves. In simple terms, they’re a tool for giving a chunk of code a unique ID.

Backing up a bit, everything on a computer is ultimately represented as a binary number — a string of zeros and ones. To generate a hash, we take this binary number and dump it through a hash function.

Here’s an example. Suppose we take a number and divide it by two. But instead of looking at the quotient, we look at the remainder. There are two possible outcomes. If the number is odd, the remainder will be 1. Or if the number is even, the remainder will be 0. This operation is a simple type of hash function. It takes a number that could be arbitrarily large, and collapses it into a single-digit hash — in this case, either ‘0’ or ‘1’.

So that’s how hashes work in principle. In practice, hash functions are complicated, and their outputted hashes are long. And that’s because their goal is to give code a unique ID.

(My example hash function gives half of all numbers the same hash, leading to numerous ‘hash collisions’. Longer hashes — say 256 bytes — vastly decrease the likelihood of such collisions, meaning we can treat the generated hashes as ‘practically unique’.)

Looking at hashes, Dolstra realized that they offer a warehouse-grade tool for tracking software. For starters, hashes provide a way to differentiate between apps. If we take Firefox code and dump it through a hash function, it will receive a different hash than if we did the same thing with LibreOffice code.

Better still, hashes can track different versions of the same program. If we take code for Firefox 121 and dump it through a hash function, it will get a different hash than code for Firefox 120.

Finally, hashes can track differences in the entire build environment that goes into making an app. So if we use GCC version 13.2 to compile code for Firefox 121, when we dump the program through a hash function, we’ll get a different hash than if we used GCC version 12.3 to do the same task.

In short, hashes are the ultimate software part number — an ID that summarizes how each and every program was made. So with hashes, Dolstra realized, we have a warehouse-grade design fit for 21st-century software management. But this warehouse came with a catch. To build it, Dolstra had to nix the Unix design and rebuild something from the ground up. Fittingly, he called the result ‘Nix’.⁴

The Nix warehouse

To understand Dolstra’s Nix warehouse, we need to start with the legacy Unix design. In a standard Unix-like system, programs are dumped in the folder ‘/usr/bin’. So an installation of Firefox would live here:

As more programs (and their dependencies) get shoved into ‘/usr/bin’, entropy does its work, and we’re left with an epic mess. Looking at this mess, Dolstra decided to nuke it and start fresh. In his ‘Nix’ system, programs would live in the directory ‘/nix/store’.

Now at first glance, this directory is simply ‘/usr/bin’ wearing different clothing. However, when we look inside ‘/nix/store’, things are different. Instead of a desktop piled high with unorganized documents, we see a software warehouse, meticulously organized with hashes.

Here’s an example. In Nix, an installation of Firefox might live here:

Starting from the left, we see the outside of the warehouse — the folder ‘/nix/store’. As we move inside the warehouse, we see the software stacks, organized by hash. Now since the hash is cryptic, it’s followed by a human-friendly label — the software name and version number (in this case, Firefox version 121.0).

With this hash-labeled directory, we have a unique shelf in our software warehouse — a shelf where this particular version of Firefox gets installed. But unlike a ‘container’, this Nix shelf doesn’t hold all of Firefox’s dependencies. Instead, those dependencies get their own shelf associated with a different hash. So if another program wants to share the same dependency, it can do so. Alternatively, if there’s a dependency clash, Nix creates two (or more) shelves — one for each version of the dependency.

Speaking of versions, when software gets updated, it gets a new shelf in the Nix warehouse. For example, suppose we update Firefox from version 121 to version 122. Nix gives this updated version an entirely new hashed shelf. So the old version lives here:

And the new version lives here:

From this warehouse design, many software-management problems simply vanish. For starters, dependency management becomes a non-issue. Since each version of each program gets its own shelf in the Nix warehouse, dependency clashes are impossible. As a consequence, software can be kept up to date without fear that it will break. And unlike containers, the Nix warehouse lets programs share resources.

The Nix warehouse also allows for easy rollbacks. Since new versions of a program get a new shelf, the old shelf still exists, should we need it. In other words, if software breaks, we can easily roll back to a previous version. In fact, with the Nix warehouse, we can easily rollback the entire operating system.

Finally, the Nix warehouse makes the whole operating system deterministic. That’s because the hash system allows Nix to maintain an inventory of our software — a database that describes the programs we’ve got installed and the complex ways this software shares dependencies. Because the Nix warehouse is database driven, we can deterministically change its contents with ease. We simply order a new software inventory, and Nix takes care of the rest, purging the old warehouse and refilling it with fresh inventory.

I could go on gushing about Nix, but you get the point. When we replace the messy-desk Unix design with an industrial-grade warehouse, many logistical problems with managing software simply go away.

Slow-motion reverse

When it comes to Dolstra’s Nix design, I find two aspects fascinating.

The first is its simple elegance. The Nix warehouse is based on an organizational regime that in hindsight, is utterly obvious. Of course each version of each program should get its own warehouse shelf, labeled with a unique ID. Actual warehouses have been organized this way for years. So it only makes sense that computer operating systems should adopt the same regime. And yet, it took until the early 2000s for a programmer to have this idea. Such is the sway of technological lock in.

And that brings me to the second fascinating feature of Nix, which is its glacial pace of adoption. The Nix design has been around since 2003. But its only in the last few years that this design has received significant traction.

Again, this slow road to adoption is typical of technological lock in. When you’ve got an established way of doing things, people are biased towards the status quo. Sure, Nix alleviates a huge number of logistical problems with how software gets managed. But to get these advantages, you’ve got to abandon your current workflow. Understandably, developers have been conservative about taking this leap. And so they’ve made do with a series of hacks that retained the legacy Unix design.

Today, though, the hack cascade is getting comical. Companies pay big money to have their servers kept ‘stable’ yet as ‘up-to-date’ as possible. There are all kinds of ‘compliance’ certifications. Heck, there are even copycats who certify that their (cheaper) copy-cat system is ‘bug-for-bug’ compliant with the name-brand certification.

The funny thing is that this hack cascade is entirely unnecessary. With Nix, there is a free way to build a deterministic, exquisitely organized warehouse for managing your entire software stack.

Fortunately, in 2024 — almost two decades after Dolstra published his thesis — people are taking notice of the Nix design. In short, we’re starting to reverse out of the Unix cul-de-sac. Hopefully, we’ll soon get on a new road, and the Nix design will become invisibly ubiquitous — like the wheels on your car.

---

Support this blog

Hi folks. I’m a crowdfunded scientist who shares all of his (painstaking) research for free. If you think my work has value, consider becoming a supporter.

---

Stay updated

Sign up to get email updates from this blog.

---

This work is licensed under a Creative Commons Attribution 4.0 License. You can use/share it anyway you want, provided you attribute it to me (Blair Fix) and link to Economics from the Top Down.

---

Nix for newbies

Perhaps the most confusing thing about Nix is that the term ‘Nix’ stands for three things:

1. ‘Nix’, the package manager;
2. ‘Nix’, the operating system;
3. ‘Nix’, the programming language.

Looking at Dolstra’s thesis, we can see that Nix started life as a warehouse-grade package manager that organized software according to cryptographic hashes. This was Dolstra’s ‘big idea’, and it informed everything that followed.

Soon after the Nix package manager was created, Dolstra and his collaborators realized that they could use it for more than managing software; they could use it to manage the whole operating system. Thus was born NixOS.

Finally, the idea behind the Nix package manager was that it made software management deterministic: Nix built and deployed exactly (and only) what you told it to build and deploy. This meant that not only was Nix a package manager for end users, it also needed to manage the whole build environment. And to do that, Nix needed a fully formed programming language. Thus was born the Nix language.

In my opinion, the best way to start using Nix is as follows. Begin by installing the Nix package manager on a distribution of your choosing. (It works on most Linux distributions, MacOS, and WSL on Windows.)

Once you’re comfortable with the package manager, test out NixOS by using the graphical installer.

Finally, when you’ve got a working version of NixOS, you can start toying with the Nix language by manipulating your configuration file.

Here are some resources that will help your Nix journey:

• Zero to Nix: perhaps the best lay introduction to Nix.
• Nix Wiki: more details about Nix that are (mostly) readable for newbies.
• Nix Manual: exhaustive documentation that you’re not going to read from cover to cover. Search the manual if you have questions/problems.

Also, the Linux Unplugged podcast is doing an excellent job covering Nix developments. They’re the folks who got me interested in Nix.

How Unix conquered the world

How did Unix become the foundation of modern computing? Answer: through an improbable, winding path.

In the late 1990s, it looked like Windows would be the world’s default OS. Today, that’s changed, largely because of two technologies: smartphones and the internet. The major smartphone operating systems — Android and IOS — are both Unix-based. And the internet? Well, it runs almost entirely on Linux — an open-source implementation of Unix.

That’s the one-paragraph story of how Unix conquered the world. If you want the long version, read on.

A monopolist’s lab

The Unix story starts with the telephone company AT&T, which for much of the 20th century, operated a government-supported monopoly over American telecommunications.

From this monopoly, AT&T was able to extract what Michael Riordan calls a “built-in ‘R&D tax’ on telephone service”. For every dollar of revenue that AT&T taxed from Americans, it sent about 2 cents to its R&D wings — Bell Labs and Western Electric. From this massive research funding, many good things sprouted, including Unix (designed at Bell Labs).

The historical backdrop is that prior to Unix, operating systems tended to be purpose-built tools constructed using low-level assembly language. But at Bell Labs, Dennis Ritchie had invented a higher-level programming language called C. Programmer Ken Thomson then decided to use this language to build an operating system. He called the result ‘Unix’.

The philosophy behind Unix was that programs should be small and simple. If you had a more complex task, you’d then chain these simple programs together via pipes. It was a brilliant design, then and now.

GNU/Linux

Born in a monopolists’ research lab in the 1970s, by the 1980s, Unix was being sold by its corporate overlord, AT&T. The target market was business mainframes. And the target price was … high. And so hackers who liked the Unix design, but didn’t like its proprietary model, started looking for alternatives.

At MIT, a group of programmers (led by Richard Stallman) set out on a ground-up reimplementation of Unix. They called the project ‘GNU’ — self-referentially short for ‘GNU’s Not Unix’. By the early 1990s, the GNU project had rewritten almost all of the Unix code, but still lacked a ‘kernel’ — the core piece of software that tells an OS how to communicate with hardware.

That’s where Finnish programmer Linus Torvalds enters the picture. In 1991, he created a Unix-like kernel that he called ‘Linux’. The Linux kernel was soon combined with the GNU code to create the GNU/Linux operating system.⁵

Initially a bit player in a Windows dominated world, Linux became popular largely because of the internet. As server farms got erected, businesses balked at paying Windows licensing fees for each machine. With Linux, they had an operating system that could do a better job than Windows … and the entire system was free.

Understandably, Linux took over the server market. And so today, it is the foundation of the internet. Oh, and when internet monopolist Google decided to make a phone operating system (Android) on which to sell its wares, it settled on the Linux kernel as the main driver. To summarize, the internet runs on a Unix-like design, as does every device powered by Android.

BSD-intosh

We’re not done with our winding path to Unix dominance. Backing up a bit, there’s a fork in the road that leads to modern-day Apple.

The fork starts in Berkeley, California. In the 1980s, the University of California Berkeley had acquired a license for Unix source code. The idea was that this code would be used solely for ‘teaching purposes’. But of course, Berkeley hackers immediately started playing with the code and building new things. And like lefty student-groups do, these hackers released their code under a permissive license. The result was called the ‘Berkeley Software Distribution’, or BSD.

As monopolists do, AT&T later sued BSD for copyright infringement. AT&T eventually lost the case. But even if it hadn’t, the lawsuit was largely futile. By the early 1990s, BSD code had escaped captivity and was being used by many projects. One of these projects was called NeXTSTEP, which was a new BSD-based graphical OS.

As it happens, NeXTSTEP was being developed by NeXT, a company founded by Steve Jobs in 1985, after he’d been forced out of Apple. Fast forward to the late 1990s, and Apple was floundering. Looking to reinvigorate its brand, Apple bought NeXT in 1997. Within months of the purchase, Steve Jobs was back at the Apple helm, and NeXTSTEP became the basis of new versions of MacOS, and later, IOS.

To recap this winding path, Unix gave rise to BSD, which was the foundation of NeXTSTEP, which became the basis of MacOS and IOS. All of which is to say that today, every Apple device is powered by a Unix design.

From bit player to bohemeth

Returning to the present, the computing world looks much different than it did in the 1990s. While Windows still dominates the desktop market, it is completely absent from the phone and server market. And so by a conservative estimate, about 70% of the world’s devices are powered by a Unix-like design. And that means Unix design flaws are locked into most of our computers.

Notes

Further reading

Dolstra, E. (2006). The purely functional software deployment model. https://edolstra.github.io/pubs/phd-thesis.pdf

Dolstra, E., De Jonge, M., Visser, E., et al. (2004). Nix: A safe and policy-free system for software deployment. LISA, 4, 79–92. https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf