Reasoning About Docker

· cyclicircuit's blog

A supplementary article for my series about self-hosting which gives us a mental model for reasoning about docker in the context of self-hosting.

Table of Contents

This post ams to explain what docker is, the basics of how it works, and how to reason about it from a self-hosting perspective. This may seem a little bit strange, but the content of this article may contradict minor points of what you already knowabout docker, this is on purpose because we want to gain an intuitive, rather than purely technical, understanding. The goal is not so much to teach precisely accurate information about docker, but rather to provide a way to reason about it effectively in a self-hosting context. Feel free to skip this article if you believe you have a strong understanding of docker.

This is a very long article, running close to 3,000 words, so its probably best consumed in pieces. I did my best to break it down. Here is what you can expect to learn:

  1. Firstly, we're going to look at the world before Docker, and compare and contrast Docker to Virtual Machines, which are an intuitively similar technology despite being radically different.
  2. Once we understand the problems that Docker can solve for us, we're going to explore what underlying concepts in Linux make Docker possible (and, coincidentally, why it only runs on Linux).
  3. After that, we're going to explore how Docker Images are defined by Dockerfiles and their structure, including elements like the ability to build Docker Images on top of one another.
  4. We're going to learn about the Docker file system, and why its a huge advantage that its read-only, as well as understanding what a Docker Image is, and what volume mappings are.
  5. We're going to learn what a Docker Container is, building on top of all the concepts above.
  6. Finally, we're going to turn everything we understand in theory, into an actual configuration file, called a docker compose or otherwise known as a stack which demonstrates all the key concepts we've learned in a real example.

The World Before Docker: What Problem Does Docker Solve? #

Using VMs to Submit School Assignments #

Imagine for a moment, that you're a university computer science student, and you're writing a complex project. In my case, it was a custom cryptographic algorithm written in C for my Cryptography course. You've written the assignment, and it works, but the course does not have strict requirements about the runtime of the assignment and your code requires specific libraries to compile properly. You wrote the code for the assignment on a specific version of Ubuntu which has those libraries installed.

What do you do when it comes time to submit the assignment? You have two basic approaches:

  1. You can write up complex, detailed instructions for the instructor to follow to make sure their Ubuntu machine is set up precisely the same way, compilation instructions to make sure they compile it precisely the right way, and finally runtime instructions appropriate to the assignment.
  2. You can just package up a VM with the code, libraries, and all the dependencies, and submit the VM (with instructions on how to run the code that is already present, compiled, and installed). You can probably guess that I went with option 2.

This is a phenomenal use case for Virtual Machines. A virtual machine is essentially guaranteed to have all the right compile-time and runtime dependencies for a given bit of code. You can totally deploy software by simply configuring a VM with the software and uploading that.

This eliminates a LOT of problems in enterprise and "cloud" software. For one thing, as a cloud services provider, you no longer have to worry about running god-only-knows-what on your precious hardware, you can provide a basic VM interface and let them run everything themselves (this is what a virtual private server is BTW). However, despite their advantages, VMs have a lot of issues. Critically, VMs are expensive to run relative to "bare metal" software. Because VM instructions are run on a host operating system and then translated to local hardware instructions using a Hypervisor, a VM is computationally at least twice as expensive as running the same software on the machine.

In some applications, particularly where security is paramount, this is an acceptable cost, but in many cases it is not. This is the problem that docker solves. One way to reason about docker is that a docker container is kind of like a VM, but without a hypervisor, meaning that code is running on the host operating system, but docker manages all the other runtime dependencies for us. Its kind of like a half-way measure to a VM, which has most (but not all) of the isolation advantages, without the extreme computational cost.

How does Docker Work (Basics) #

What Makes Docker Possible? #

Docker only runs on Linux. Even when its running on MacOS or Windows, its actually running in a Linux VM. Why?

Well, Linux has one really important design property that makes Docker work - everything is a file. You probably experienced this already if you've ever had to format a disk in Linux, you might've worked with file paths like /dev/sda. /dev is a virtual file system where the kernel provides an interface for you which looks like a file system to make it easier to use. Another example of this can be seen in /proc where data on currently running processes can be viewed by exploring the files and directories (this is how programs like htop work BTW, they're just scanning /proc and making it look pretty).

There's no registry, no crazy database structures. Everything is a file. In theory, if you copied all the files necessary for a database from one server onto another with the same OS, it will just work.

OK, but how does this help with Docker? Enter chroot - a program that launches another program within a directory which is presented as the entire file system for that program. So for example, if we launch the program ls and tell it to list all files in / within a chroot with the directory /var/run, the ls program thinks its listing the contents of / but its actually listing the contents of /var/run. Its a virtual file system, much like a virtual machine's instructions are interpreted to local hardware, a chroot file system is mapping a virtual file system to a real one. The difference is that this form of mapping is very cheap computationally, it only makes a difference when a file is initially accessed.

Docker as a File System #

The above is really profound. It may not look like much, but imagine that I told you that you need to run two slightly different versions of PostgreSQL on the same machine. If you tried to install 13.2 and 13.3 on the same machine using apt they would conflict with each other during package installation, and even if you managed to overcome that, they'd end up conflicting with each other's configuration files. However, by setting up a chroot for each one, and including all the necessary dependencies in each chroot, you could totally run two different versions of PostgreSQL in a way that would not conflict with each other.

OK, chroot seems great, but what is Docker? Well, its basically a chroot. Unfortunately, not everything is a file (I know, I'm sorry, I lied). There are other things other than files necessary to make complex software work. For example, there are environment variables, user associations, and network elements like ports. Well, that's what docker is - its chroot with all the other stuff, like networking, ports, and user mapping as well. There are, however, a few other important concepts that make docker work well.

Docker Images & Dockerfile #

One of the issues that you might have thought of when presented with the VM example above, is that its actually really hard to send an entire VM to a professor to grade - its quite large. I think the smallest I could get the VM to be was like 4GiB. What if we could write a script instead, that would tell the computer:

get the Ubuntu 24.04 LTS version ISO image

install all the dependencies with apt

download the source code from github

compile the source code

when someone wants to run this, run the code compiled above

This would still be consistent, and it would take up WAY less than 4GiB in transit, right?

Good news!

What I described above is a Dockerfile. In many different github repos, you might find a file called Dockerfile which does precisely this. Let's take a look at an example:

 1#FROM - Image to start building on.
 2FROM ubuntu:14.04
 3
 4#MAINTAINER - Identifies the maintainer of the dockerfile.
 5MAINTAINER ian.miell@gmail.com
 6
 7#RUN - Runs a command in the container
 8RUN echo "Hello world" > /tmp/hello_world.txt
 9
10#CMD - Identifies the command that should be used by default when running the image as a container.
11CMD ["cat", "/tmp/hello_world.txt"]

What's happening here? Well, we tell it to download Ubuntu 14.04, we run a command to print the words "Hello world" to a file called /tmp/hello_world.txt, and finally, we tell the computer that if someone wants to run this, we print out the contents of the aforementioned file.

When you "compile" a Dockerfile, what you get is a docker image. The docker image is essentially the aforementioned chroot file system, with some other metadata, packaged into a single file. You can send this docker image around and running this docker image will be consistent on any system with the same kernel version since the kernel is the only thing not included in the docker image.

Inheritance #

OK, the above is pretty cool, right? But how can we make it even easier for us? One cool thing that docker has, is the ability to take one docker image, and essentially, build on top of it!.

Let's look at an example: This is the Dockerfile for microbin, one of the applications that we will be setting up in the blog series: https://github.com/szabodanika/microbin/blob/master/Dockerfile

Note that it starts with the following:

1FROM rust:latest as build
2
3WORKDIR /app
4
5RUN \
6  DEBIAN_FRONTEND=noninteractive \
7  apt-get update &&\
8  apt-get -y install ca-certificates tzdata

There's not an ubuntu:14.04 or anything similar there, what the hell is rust:latest??

rust is a Dockerfile/Docker Image of its own. The glory of open-source software is, of course, that we can see everything. In this case, the rust docker image is available over at DockerHub: https://hub.docker.com/_/rust and we can follow the links there to see the rust Dockerfile:

1FROM buildpack-deps:bookworm
2
3LABEL org.opencontainers.image.source=https://github.com/rust-lang/docker-rust
4
5ENV RUSTUP_HOME=/usr/local/rustup \
6    CARGO_HOME=/usr/local/cargo \
7    PATH=/usr/local/cargo/bin:$PATH \
8    RUST_VERSION=1.87.0

Here we see that the rust image is based on buildpack-deps:bookworm docker image, which is of course available on DockerHub as well. The dependency chain ultimately looks something like this:

Docker is a Contract #

I have a secret to tell: I did not like Docker when it first came out. I thought it was a stupid idea.

What made me change my mind? Years after Docker came out, my friend pointed out something really important:

The key advantage of docker is that it isolates you from the idiocy of whoever wrote the code. Its an enforceable contract that the software author has to abide by for their software to work. The contract promises that if they are provided a specific volume mount, a specific set of ports, and a specific set of environment variables, then their software will function as expected.

This is really really cool, because once we have a consistent interface to something that everyone is forced to respect, we can build other systems on top. This means that we can build awesome things like Kubernetes on top of docker by treating docker images and containers as "objects" in memory and on the file system. We can move them around, start them up, shut them down, and they continue to run consistently.

But how do we actually guarantee this? After all, one of the issues you may have considered above when I told you to just run PostgreSQL in two different chroots is that you have no idea where all those PostgreSQL files are. After all, the configuration files are in one place, the logs are in another (if they're not in systemd), and the actual data being stored by the application is in a third. Add to this a bunch of user configuration nonsense and we get a really confusing list of files which might even change from one version of the software to another.

The core problem here is that programmers, generally, expect to have access to the whole file system (modulo some permissions stuff) and can store stateful information anywhere they want. The next profound element of Docker that makes it work so well is that the docker container's file system (except for mounted volumes) read-only.

Initially that seems pretty weird. If the file-system is read only, what's the point? Well, it forces programmers to only write information to the specific mapped volumes. If they write information anywhere else, it will not be available after the container is restarted. This forces a certain amount of consistency in these containers, it makes images behave consistently, and it enables all the key innovations that have been built on top of docker.

Volumes #

The last little bit left to understand in this section is volume mapping. Specifically, this is how we actually retain state for a docker container. We map a specific part of the file system on the host machine, to a different part of the file system of the running docker container. That's really it.

There is, just like with everything else, something profound about this though. Because, by explicitly listing all the volume mappings, we can know where all the state of a given docker container is. Why is this so cool? Because it allows really effective backups. By backing up the specific Docker Image, the configuration file for docker itself (see the Docker Compose Stacks section below), we know for a fact that we have the entire state of the running docker container saved and backed up. There is no need to worry that we missed something, because everything is listed explicitly.

Docker Containers #

To re-iterate: A Dockerfile is a text-file written in a special language that defines the key properties required to build a Docker Image. The Docker Image, in turn, is the "frozen", read-only file system compressed into a single file, that is used to provide all the runtime dependencies that can be expressed as files for a given piece of running software. Docker Images can be built on top of each other through relationships defined in the Dockerfile.

Now that we understand the relationship between Dockerfiles and Docker Images, we need to understand precisely what a Docker Container is (again, from our limited and potentially inaccurate self-hosting perspective).

The Docker Container is the actual thing that runs the software. Its the process that is launched inside the aforementioned chroot of the docker image with all the other key stuff like networking configuration, port mapping, volume mappings, and environment variables.

Docker Compose Stacks #

So how do we combine all the information above, into something concrete? Cause the above is probably a LOT of description and talk, not a lot of actual configuration that does the thing.

Some of you, may have recognized an inherent problem in the above examples: Sure, its great that you can spin up containers for things like PostgreSQL or MicroBin, but what if I have a "real" service, something like Paperless or something that needs PostgreSQL, Redis, and RabbitMQ to function properly? These are not unreasonable requirements, the aforementioned services are extremely efficient database, caching, and queueing systems that you don't want to replicate in your software, its WAY better to use those as pre-packages services. If I have to set up my own PostgreSQL docker container, and have all my web apps connect to it, that's actually pretty damn complicated.

This is where docker compose comes into play. The idea is that most applications of any appreciable complexity are going to need closely associated services. For this, we have the docker compose specification. The idea is that we can combine the configuration for several docker containers into a single file, and run them more-or-less as one unit. The PostgreSQL container, for example, would exist purely to support our paperless instance, and if we want to spin up another service that depends on PostgreSQL, well they can have their own, completely separate database too in a completely separate network and everything.

Let's look at an example to understand how it works!

 1services:
 2  db:
 3    image: postgres:alpine
 4    restart: always
 5    volumes:
 6      - /var/lib/docker-apps/nextcloud/db:/var/lib/postgresql/data:Z
 7    environment:
 8      - POSTGRES_PASSWORD={{ nextcloud.db_passwd }}
 9      - POSTGRES_DB=nextcloud
10      - POSTGRES_USER=nextcloud
11
12
13  redis:
14    image: redis:alpine
15    restart: always
16
17  app:
18    image: nextcloud:apache
19    restart: always
20    ports:
21      - 127.0.0.1:{{ nextcloud.port }}:80
22    volumes:
23      - /var/lib/docker-apps/nextcloud/data:/var/www/html:z
24    environment:
25      - POSTGRES_HOST=db
26      - REDIS_HOST=redis
27      - POSTGRES_PASSWORD={{ nextcloud.db_passwd }}
28      - POSTGRES_DB=nextcloud
29      - POSTGRES_USER=nextcloud
30      - NEXTCLOUD_TRUSTED_DOMAINS={{ nextcloud.hostname }}
31      - OVERWRITECLIURL=https://{{ nextcloud.hostname }}
32      - OVERWRITEPROTOCOL=https
33      - OVERWRITEHOST={{ nextcloud.hostname }}
34    depends_on:
35      - db
36      - redis
37
38  cron:
39    image: nextcloud:apache
40    restart: always
41    volumes:
42      - /var/lib/docker-apps/nextcloud/data:/var/www/html:z
43    entrypoint: /cron.sh
44    depends_on:
45      - db
46      - redis

This happens to be a template I used to use for NextCloud. I would configure these with Ansible, which is why you see all these Jinja2-style variables like {{ nextcloud.port }} but you can mostly ignore those, they're stand-ins for real values that would be there.

Let's go through this piece by piece and understand what's happening:

 1services:
 2  db:
 3    image: postgres:alpine
 4    restart: always
 5    volumes:
 6      - /var/lib/docker-apps/nextcloud/db:/var/lib/postgresql/data:Z
 7    environment:
 8      - POSTGRES_PASSWORD={{ nextcloud.db_passwd }}
 9      - POSTGRES_DB=nextcloud
10      - POSTGRES_USER=nextcloud

A docker compose file typically begins with a services section, which defines all the docker containers being run together as one unit. The first, in this case, is the service named db, which is not very descriptive until we see the line image: postgres:alpine. This tells us that we're using the postgres:alpine image, so this is a PostgreSQL database. From there, everything basically should make sense with respect to everything we discussed before:

1  redis:
2    image: redis:alpine
3    restart: always

Here we have our redis container, by this point what's happening here should be obvious, we're creating the container based on the redis:alpine image, and we say it should always be restarted. Don't even need much else.

 1  app:
 2    image: nextcloud:apache
 3    restart: always
 4    ports:
 5      - 127.0.0.1:{{ nextcloud.port }}:80
 6    volumes:
 7      - /var/lib/docker-apps/nextcloud/data:/var/www/html:z
 8    environment:
 9      - POSTGRES_HOST=db
10      - REDIS_HOST=redis
11      - POSTGRES_PASSWORD={{ nextcloud.db_passwd }}
12      - POSTGRES_DB=nextcloud
13      - POSTGRES_USER=nextcloud
14      - NEXTCLOUD_TRUSTED_DOMAINS={{ nextcloud.hostname }}
15      - OVERWRITECLIURL=https://{{ nextcloud.hostname }}
16      - OVERWRITEPROTOCOL=https
17      - OVERWRITEHOST={{ nextcloud.hostname }}
18    depends_on:
19      - db
20      - redis

Here is where we get into something interesting. This is the "main" NextCloud container definition. We see some things, like the image:, volumes:, and environment: which should already be familiar, however, there are a few new things too:

1  cron:
2    image: nextcloud:apache
3    restart: always
4    volumes:
5      - /var/lib/docker-apps/nextcloud/data:/var/www/html:z
6    entrypoint: /cron.sh
7    depends_on:
8      - db
9      - redis

And finally we have this cron thing, which is really just a nextcloud container repurposed for some specific tasks (like cron jobs). Not much new here other than the fact that we specify an entrypoint which is a script that gets run inside the container on startup instead of what would normally be defined in the Dockerfile.

Conclusion #

By the end of this article, you should have a solid foundation of how to reason about docker as a self-hoster. Note that some elements presented here may have inaccuracies, however, if you read the above thoroughly what you will find is that it is relatively easy to search for what you want to understand and dig into it for a more formal overview. Best of luck!

Next, we are going to dive into managing docker containers with dockge.

#homelab #docker
last updated: