Podman on Major Hayden

Getting podman quadlets talking to each other

major@mhtx.net (Major Hayden) — Thu, 25 Sep 2025 00:00:00 +0000

Quadlets are a handy way to manage containers using systemd unit files. Containers running via quadlets have access to the external network by default, but they don’t automatically communicate with each other like they do in a docker-compose setup. Adding networking only requires a few extra steps.

Setting up some quadlets

I often need a postgres server laying around on my local machine for quick tasks or testing something I’m working on. Lately, I’ve been focused on RAG databases and that usually involves pgvector.

The pgvector extension adds vector data types and functions to PostgreSQL, which is great for storing embeddings from machine learning models. You can search via all of the usual SQL queries that you’re used to, but pgvector adds new capabilities for searching rows based on vector similarity.

Here’s the quadlet for pgvector in ~/.config/containers/systemd/pgvector.container:

[Unit]
Description=pgvector container
After=network-online.target

[Container]
Image=docker.io/pgvector/pgvector:pg17
Volume=pgvector:/var/lib/postgresql/data
Environment=POSTGRES_USER=postgres
Environment=POSTGRES_PASSWORD=secrete
Environment=POSTGRES_DB=postgres
PublishPort=5432:5432

[Service]
Restart=unless-stopped

This gets a postgres server with pgvector up and running with a persistent volume. It’s listening on the default port 5432.

Sometimes I’m in a hurry and pgadmin4 is a quick way to poke around the database. It’s also a good example here since it needs to talk to the pgvector container. Here’s the quadlet for pgadmin4 in ~/.config/containers/systemd/pgadmin4.container:

[Unit]
Description=pgAdmin4 container
After=network-online.target

[Container]
Image=docker.io/dpage/pgadmin4:latest
Volume=pgadmin4:/var/lib/pgadmin
Environment=PGADMIN_DEFAULT_EMAIL=major@mhtx.net
Environment=PGADMIN_DEFAULT_PASSWORD=secrete
Environment=PGADMIN_CONFIG_SERVER_MODE=False
Environment=PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
PublishPort=8080:80

[Service]
Restart=unless-stopped

Awesome! Let’s reload the systemd configuration for my user account and start these containers:

systemctl --user daemon-reload
systemctl --user start pgvector
systemctl --user start pgadmin4

We can check the running containers:

> podman ps --format "table {{.ID}}\t{{.Names}}"
CONTAINER ID NAMES
b099fdaa6b18 valkey
f8ab764c299c systemd-pgadmin4
052c160fb45b systemd-pgvector

Testing communication

Let’s hop into the pgadmin4 container and see if we can connect to the pgvector database:

> podman exec -it systemd-pgadmin4 ping pgvector -c 4
ping: bad address 'pgvector'
> podman exec -it systemd-pgadmin4 ping systemd-pgvector -c 4
ping: bad address 'systemd-pgvector'

This isn’t great. There are two problems here:

The containers aren’t on the same network
I want to refer to the pgvector container as pgvector, not systemd-pgvector

Let’s fix that.

Fixing communication

Open up the ~/.config/containers/systemd/pgvector.container file and make the two changes noted below with comments:

[Unit]
Description=pgvector container
After=network-online.target

[Container]
# Use a consistent name 👇
ContainerName=pgvector
Image=docker.io/pgvector/pgvector:pg17
Volume=pgvector:/var/lib/postgresql/data
Environment=POSTGRES_USER=postgres
Environment=POSTGRES_PASSWORD=secrete
Environment=POSTGRES_DB=postgres
# Add the container to a network 👇
Network=db-network
PublishPort=5432:5432

[Service]
Restart=unless-stopped

Also open the ~/.config/containers/systemd/pgadmin4.container file and make the same network change:

[Unit]
Description=pgAdmin4 container
After=network-online.target

[Container]
Image=docker.io/dpage/pgadmin4:latest
Volume=pgadmin4:/var/lib/pgadmin
Environment=PGADMIN_DEFAULT_EMAIL=major@mhtx.net
Environment=PGADMIN_DEFAULT_PASSWORD=secrete
Environment=PGADMIN_CONFIG_SERVER_MODE=False
Environment=PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED=False
# Add the container to a network 👇
Network=db-network
PublishPort=8080:80

[Service]
Restart=unless-stopped

Create the network:

podman network create db-network

Now, reload the systemd configuration and restart the containers:

systemctl --user daemon-reload
systemctl --user restart pgvector
systemctl --user restart pgadmin4

Testing communication again

Now, let’s hop into the pgadmin4 container and see if we can connect to the pgvector database:

> podman exec -it systemd-pgadmin4 ping pgvector -c 4
PING pgvector (10.89.5.6): 56 data bytes
64 bytes from 10.89.5.6: seq=0 ttl=42 time=0.026 ms
64 bytes from 10.89.5.6: seq=1 ttl=42 time=0.036 ms
64 bytes from 10.89.5.6: seq=2 ttl=42 time=0.034 ms
64 bytes from 10.89.5.6: seq=3 ttl=42 time=0.088 ms

--- pgvector ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.026/0.046/0.088 ms

Perfect! 🎉 🎉 🎉

Extra credit

If you want to deploy your system with automation and avoid the manual network creation, you can add one extra file to your ~/.config/containers/systemd/ directory. Save this as db-network.network:

[Network]
Label=app=db-network

Automatic container updates with Podman quadlets

major@mhtx.net (Major Hayden) — Fri, 19 Sep 2025 00:00:00 +0000

Running containers at home or in production often means juggling updates across multiple services. While orchestration platforms like Kubernetes handle this automatically, what about those simple deployments on a single host?

Podman’s quadlet system integrates containers directly with systemd, and when combined with automatic updates, you get a robust solution that keeps your containers fresh without manual intervention.

Let’s explore how to set up automatic container updates using Podman quadlets on Fedora, turning container management into a hands-off operation that just works.

Setting up a basic quadlet

First, let’s create a simple quadlet for running a Valkey database service under a user account. Quadlet files for user services live in ~/.config/containers/systemd/.

Create the directory if it doesn’t exist:

mkdir -p ~/.config/containers/systemd/

Then create your quadlet file:

# ~/.config/containers/systemd/valkey.container
[Container]
ContainerName=valkey
Image=docker.io/valkey/valkey:latest
Label=io.containers.autoupdate=registry
PublishPort=16379:6379
Volume=valkey_data:/data

[Service]
Restart=always

[Install]
WantedBy=default.target

The magic happens with the Label=io.containers.autoupdate=registry line. This label tells Podman that this container should be automatically updated when a newer image is available in the registry.

After creating the file, reload systemd and start your container:

systemctl --user daemon-reload
systemctl --user start valkey.service

Your container is now running as a user systemd service! Check its status with:

systemctl --user status valkey.service
podman ps

Enabling automatic updates

Podman ships with a systemd timer that checks for container updates. The podman-auto-update.timer runs daily by default, but you need to enable it for your user:

systemctl --user enable --now podman-auto-update.timer

You can check when the next update will run:

systemctl --user list-timers podman-auto-update.timer

When the timer triggers, it runs podman auto-update, which:

Checks all containers with the io.containers.autoupdate label
Pulls newer images if available
Restarts containers with the new image
Keeps the old image in case you need to roll back

Customizing update behavior

The io.containers.autoupdate label supports different values for various update strategies:

# Always pull the latest image from the registry
Label=io.containers.autoupdate=registry

# Only update if the local image changes (useful for locally built images)
Label=io.containers.autoupdate=local

You can also customize when updates occur by creating a timer override:

systemctl --user edit podman-auto-update.timer

Add these lines to run updates every 6 hours instead of daily:

[Timer]
OnCalendar=
OnCalendar=*-*-* 00,06,12,18:00:00

Monitoring updates

Track what’s happening with your automatic updates using journalctl:

# View recent auto-update logs
journalctl --user -u podman-auto-update.service -n 50

# Follow updates in real-time
journalctl --user -u podman-auto-update.service -f

# Check a specific container's restart history
journalctl --user -u valkey.service | grep Started

How I learned to stop worrying and love the CoreOS

major@mhtx.net (Major Hayden) — Fri, 13 Oct 2023 00:00:00 +0000

It’s quite clear that I’ve been on a CoreOS blogging streak lately. I keep getting asked by people inside and outside my company about what makes CoreOS special and why I’ve switched over so many workloads to it.

The answer is pretty basic. It makes my life easier.

I’m a Dad. I’m on the PTC (Parent Teacher Club) at one of my children’s schools. I volunteer as an IT person for a non-profit. I write software. I have other time consuming hobbies, such as ham radio, reading, and becoming a longer distance runner¹.

My available time for my own IT projects is extremely limited and CoreOS plays a part in keeping that part of my life as efficient as possible.

That’s what this blog post is about!

Updates

First and foremost, I love how CoreOS does updates. I encourage you to read the docs on this topic, but here’s a short explanation:

Updates are automatically retrieved and they’re loaded into a slot.
Your system reboots into the new update but your original OS tree remains in place.
Did the update boot? Awesome. You’re good to go.
Did something break? The system reverts back to the known good tree.

In this way, it’s a lot like your smartphone.

You have full control over when a node looks for an update and how often it checks for them. Check out the Zincati docs for tons of controls over updates and reboots.

Some of mine are timed so well that I set maintenance windows with my monitoring provider when I know an update might take place. The updates come through, monitoring shuts off, the node reboots, and monitoring comes back. The nodes almost always come back before the monitoring even alerts me.

It also removes the reminders I would set for myself to update packages and run reboots. I know that my CoreOS nodes will do this automatically, so I don’t need to think about it.

Also, updates are rarely ever impactful to my workloads since all of them are running inside containers. My containers come right back up as soon as the node finishes its reboot.

Toolbox

To be fair, you can get toolbox running on lots of different Linux distributions outside of CoreOS, but that’s the first place I ever used it. Toolbox, also called toolbx, gives you a utility container on your CoreOS node for all kinds of adminsitrative and diagnostic capabilities.

You might need a certain package for diagnosting a hardware issue or you might want to install some helpful utilities for the command line. Do that in a toolbox container. Just run toolbox enter and if you’ve never created a toolbox container before, you’ll get a Fedora container that matches your CoreOS release.

But it gets better.

Toolbox automatically saves your container when you’re done with it so all of your installed packages stay there for next time. Also, these containers have seamless access to anything you have in your home directory, including sockets. You’re running inside a container, but it’s almost like you’re running on the host itself inside your home directory. You get the best of both worlds.

Don’t want Fedora? You have lots of distribution options through toolbox. Read the details on custom images to create your own!

Layering

Okay, there are those situations where you really want a package on CoreOS and toolbox might not be sufficient. My muscle memory for vim is so strong and CoreOS only comes with vi.

You have a couple of options here:

Run sudo rpm-ostree install vim, reboot, and you have vim
Run sudo rpm-ostree install --apply-live vim and you have vim right now! (And it’s there after a reboot as well.)

When a new update comes down for the base OS from CoreOS, any packages you’ve added will be layered on the base image and available after a reboot. Layering is generally chosen as a last resort option for adding packages to the system but you shouldn’t run into issues if you’re installing small utilities or command line tools.

Declarative provisioning

If you’ve provisioned Linux distributions on cloud instances in the past, you’ve likely provided metadata that cloud-init uses to provision your system. CoreOS has something that acts a lot earlier in the boot process and has more power to get things done: ignition.

There’s a handy butane file forma that you use for writing your configuration. You use the butane utility to get it into ignition format. The ignition format is highly compressed to ensure you can fit your configuration into most cloud providers’ metadata fields.

For a real example of what you can do with ignition, check out my quadlets post where I provisioned an entire Wordpress container stack using a single ignition file.

There’s lots of documentation for writing butane configuration files for common situations. It’s easy to add files, configure Wireguard, set up users, and launch containers immediately on the first boot.

Pets and cattle

CoreOS works well for systems that I only need online for a short time. These might be situations where I need to test a few containers and throw it away. There’s no OS to mess with and no updates to worry about.

It also works well for systems that I keep online for a long time. I have a few physical systems at home that run CoreOS and they’ve been extremely stable. I also have cloud instances on Hetzner, VULTR, and Digital Ocean that have run CoreOS for months without issues.

Quadlets might make me finally stop using docker-compose

major@mhtx.net (Major Hayden) — Mon, 25 Sep 2023 00:00:00 +0000

I’ve written a lot about containers on this blog. Why do I love containers so much?

They start quickly
They make your workloads portable
They disconnect your application stack from the OS that runs underneath
You can send your application through CI as a single container image
You can isolate workloads on the network and limit their resource usage much like a VM

However, I’m still addicted to docker-compose. Can podman’s quadlets change that?

Yes, I think they can.

What’s a quadlet?

Podman introduced support for quadlets in version 4.4 and it’s a simpler way of letting systemd manage your containers. There was an option in the past to have podman generate systemd unit files, but those were unwieldy and full of podman command line options inside a unit file. These unit files weren’t easy to edit or even parse with eyeballs.

Quadlets make this easier by giving you a simple ini-style file that you can easily read and edit. This blog post will include some quadlets later, but here’s an example one for Wordpress:

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Lots of the lines under [Container] should look familiar to most readers who have worked with containers before. However, there’s something new here.

Check out the AutoUpdate=registry line. This tells podman to keep your container updated on a regular basis with the upstream container registry. I’ve used watchtower in the past for this, but it requires a privileged container and it’s yet another external dependency.

Also, at the very end, you’ll see a WantedBy line. This is a great place to set up container dependencies. In this example, the container that runs caddy (a web server) can’t start until Wordpress is up and running.

So why not stick with docker-compose?

There’s no denying that docker-compose is an awesome tool. You specify the desired outcome, tell it to bring up containers, and it gets containers into the state you specified. It handles volumes, networks, and complicated configuration without a lot of legwork. The YAML files are pretty easy to read, too.

However, as with watchtower, that’s another external dependency.

My container deployments are often done at instance boot time and I don’t make too many changes afterwards. I found myself using docker-compose for the initial deployment and then I didn’t really use it again.

Why not remove it entirely and use what’s built into CoreOS already?

Quaint quadlets quickly!

Before we start, we’re going to need a few things:

An easy to read butane configuration which gets transformed into a tiny ignition configuration for CoreOS
Some quadlets
Extra system configuration
A cloud provider with CoreOS images (using VULTR for this)

I’ve packed all of these items into my quadlets-wordpress repository to make it easy. Start by looking at the config.butane file.

Let’s break it down here. First up, we add an ssh key for the default core user.

variant: fcos
version: 1.5.0
passwd:
 users:
 - name: core
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0

Next up, we enable the podman-auto-update.timer so we get container updates automatically:

storage:
 links:
 - path: /home/core/.config/systemd/user/timers.target.wants/podman-auto-update.timer
 target: /usr/lib/systemd/user/podman-auto-update.timer
 user:
 name: core
 group:
 name: core

Next is the long files section:

 files:
 # Ensure the `core` user can keep processes running after they're logged out.
 - path: /var/lib/systemd/linger/core
 mode: 0644
 
 # Allow caddy to listen on 80 and 443.
 # Allow it to ask for bigger network buffers, too.
 - path: /etc/sysctl.d/90-caddy.conf
 contents:
 inline: |
 net.ipv4.ip_unprivileged_port_start = 80
 net.core.rmem_max=2500000
 net.core.wmem_max=2500000

 # Set up an an environment file that containers can read to configure themselves.
 - path: /home/core/.config/containers/containers-environment
 contents:
 inline: |
 MYSQL_DATABASE=wordpress
 MYSQL_USER=wordpress
 MYSQL_ROOT_PASSWORD=mariadb-needs-a-secure-password
 MYSQL_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_HOST=mariadb
 WORDPRESS_DB_USER=wordpress
 WORDPRESS_DB_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_NAME=wordpress
 mode: 0644

 # Deploy the caddy configuration file from the repository.
 - path: /home/core/.config/caddy/Caddyfile
 contents:
 local: caddy/Caddyfile
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add some named volumes for caddy and wordpress.
 - path: /home/core/.config/containers/systemd/caddy-config.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/caddy-data.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/wordpress.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core

 # Create a network for all the containers to use and enable the
 # DNS plugin. This allows containers to find each other using
 # the container names.
 - path: /home/core/.config/containers/systemd/wordpress.network
 contents:
 inline: |
 [Network]
 DisableDNS=false
 Internal=false
 user:
 name: core
 group:
 name: core

 # Add the wordpress container.
 - path: /home/core/.config/containers/systemd/wordpress.container
 contents:
 local: quadlets/wordpress.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the MariaDB container.
 - path: /home/core/.config/containers/systemd/mariadb.container
 contents:
 local: quadlets/mariadb.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the caddy container.
 - path: /home/core/.config/containers/systemd/caddy.container
 contents:
 local: quadlets/caddy.container
 mode: 0644
 user:
 name: core
 group:
 name: core

The Caddyfile is also in the repository and will be deployed by the butane configuration shown above.

We can go through each quadlet in detail. First up is MariaDB. We tell systemd that the wordpress container will want to have this one started first.

[Unit]
Description=MariaDB Quadlet

[Container]
Image=docker.io/library/mariadb:11
ContainerName=mariadb
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=mariadb.volume:/var/lib/mysql
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=wordpress.service multi-user.target default.target

The wordpress quadlet is much the same as the MariaDB one, but we tell systemd that caddy will want wordpress started first.

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Finally, the caddy quadlet contains four volumes and some published ports. These ports will be published to the container host. Also, you’ll note that the wordpress volume is mounted here, too. This is because caddy can serve static files much faster than wordpress can.

[Unit]
Description=Caddy Quadlet

[Container]
Image=docker.io/library/caddy:latest
ContainerName=caddy
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=caddy-data.volume:/data
Volume=caddy-config.volume:/config
Volume=/home/core/.config/caddy/Caddyfile:/etc/caddy/Caddyfile:Z
Volume=wordpress.volume:/var/www/html
PublishPort=80:80
PublishPort=443:443
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

Launch the quadlets

There’s a launch script that ships this configuration to VULTR and launches a CoreOS instance:

#!/bin/bash
# This command starts up a CoreOS instance on Vultr using the vultr-cli
vultr-cli instance create \
 --os 391 \
 --plan vhp-1c-1gb-amd \
 --region dfw \
 --notify true \
 --ipv6 true \
 -u "$(butane --files-dir . config.butane)" \
 -l "coreos-$(date "+%s")"

To launch an instance, get your VULTR API key first. Then install vultr-cli and butane:

$ sudo dnf -y install butane vultr-cli

After launch, check to see what your containers are doing:

[core@vultr ~]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afa2d6501593 docker.io/library/caddy:latest caddy run --confi... 54 seconds ago Up 53 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp caddy
460426f39e6c docker.io/library/mariadb:11 mariadbd 35 seconds ago Up 35 seconds mariadb
92ece6538d5a docker.io/library/wordpress:fpm php-fpm 28 seconds ago Up 29 seconds wordpress

We should be able to talk to wordpress through caddy on port 80:

[core@vultr ~]$ curl -si http://localhost/wp-admin/install.php | head -n 25
HTTP/1.1 200 OK
Cache-Control: no-cache, must-revalidate, max-age=0
Content-Type: text/html; charset=utf-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Server: Caddy
X-Powered-By: PHP/8.0.30
Date: Mon, 25 Sep 2023 21:43:40 GMT
Transfer-Encoding: chunked

<!DOCTYPE html>
<html lang="en-US" xml:lang="en-US">
<head>
	<meta name="viewport" content="width=device-width" />
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
	<meta name="robots" content="noindex,nofollow" />
	<title>WordPress &rsaquo; Installation</title>
	<link rel='stylesheet' id='dashicons-css' href='http://localhost/wp-includes/css/dashicons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='http://localhost/wp-includes/css/buttons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='http://localhost/wp-admin/css/forms.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='http://localhost/wp-admin/css/l10n.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='http://localhost/wp-admin/css/install.min.css?ver=6.3.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui language-chooser">
<p id="logo">WordPress</p>

Awesome! 🎉

Managing containers

Containers will automatically update on a schedule and you can check the timer:

[core@vultr ~]$ systemctl status --user podman-auto-update.timer
● podman-auto-update.timer - Podman auto-update timer
 Loaded: loaded (/usr/lib/systemd/user/podman-auto-update.timer; enabled; preset: disabled)
 Active: active (waiting) since Mon 2023-09-25 21:41:31 UTC; 3min 14s ago
 Trigger: Tue 2023-09-26 00:04:46 UTC; 2h 20min left
 Triggers: ● podman-auto-update.service

Sep 25 21:41:31 vultr.guest systemd[1786]: Started podman-auto-update.timer - Podman auto-update timer.

Quadlets are just regular systemd units:

[core@vultr ~]$ systemctl list-units --user | grep -i Quadlet
 caddy.service loaded active running Caddy Quadlet
 mariadb.service loaded active running MariaDB Quadlet
 wordpress.service loaded active running Wordpress Quadlet

As an example, you can make changes to caddy’s config file and restart it easily:

[core@vultr ~]$ systemctl restart --user caddy
[core@vultr ~]$ systemctl status --user caddy
● caddy.service - Caddy Quadlet
 Loaded: loaded (/var/home/core/.config/containers/systemd/caddy.container; generated)
 Drop-In: /usr/lib/systemd/user/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Mon 2023-09-25 21:46:28 UTC; 5s ago
 Main PID: 2652 (conmon)
 Tasks: 18 (limit: 1023)
 Memory: 15.1M
 CPU: 207ms

If you need to change a quadlet’s configuration, just open up the configuration file in your favorite editor under ~/.config/containers/systemd, reload systemd, and restart the container:

$ vi ~/.config/containers/systemd/caddy.container

--- make your edits and save the quadlet configuration ---

$ systemctl daemon-reload --user
$ systemctl restart --user caddy

Enjoy!

Fixing a ghost database migration failure

major@mhtx.net (Major Hayden) — Thu, 31 Aug 2023 00:00:00 +0000

I love learning about the behind the scenes aspects of just about everything. I do ham radio, I self-host lots of my personal infrastructure, and I’ve been learning more about the math behind the stock market for the last year or two.

That led me to start a blog on Ghost to share my findings with others. I started Theta Nerd¹ earlier this summer.

My deployment looked great when I started! Everything was automatically updated with watchtower and running with docker-compose on Fedora CoreOS. (Click these links to read the posts on both topics!)

However, I woke up one morning to my monitoring going off and my site was down. 😱

Why is the site down?

Anyone who has worked in IT knows this sinking feeling. Something is down, you don’t know why, and you suspect the worst possible scenarios.

The instance hosting the blog was online and responsive, so I started digging into the logs with docker-compose logs. I suddenly found a wall of text in the logs for the Ghost container:

[2023-08-03 11:10:16] INFO Adding members.email_disabled column
[2023-08-03 11:10:16] INFO Setting email_disabled to true for all members that have their email on the suppression list
[2023-08-03 11:10:16] INFO Setting nullable: stripe_products.product_id
[2023-08-03 11:10:16] INFO Adding table: donation_payment_events
[2023-08-03 11:10:16] INFO Rolling back: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible..
[2023-08-03 11:10:16] INFO Dropping table: donation_payment_events
[2023-08-03 11:10:16] INFO Dropping nullable: stripe_products.product_id with foreign keys disabled
[2023-08-03 11:10:16] INFO Setting email_disabled to false for all members
[2023-08-03 11:10:16] INFO Removing members.email_disabled column
[2023-08-03 11:10:16] INFO Rollback was successful.
[2023-08-03 11:10:16] ERROR alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.

alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
{"config":{"transaction":false},"name":"2023-07-27-11-47-49-create-donation-events.js"}
"Error occurred while executing the following migration: 2023-07-27-11-47-49-create-donation-events.js"
Error ID:
 300
Error Code: 
 ER_FK_INCOMPATIBLE_COLUMNS
----------------------------------------
Error: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
 at /var/lib/ghost/versions/5.57.2/node_modules/knex-migrator/lib/index.js:1032:19
 at Packet.asError (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packets/packet.js:728:17)
 at Query.execute (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/commands/command.js:29:26)
 at Connection.handlePacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:478:34)
 at PacketParser.onPacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:97:12)
 at PacketParser.executeStart (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packet_parser.js:75:16)
 at Socket.<anonymous> (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:104:25)
 at Socket.emit (node:events:513:28)
 at addChunk (node:internal/streams/readable:315:12)
 at readableAddChunk (node:internal/streams/readable:289:9)
 at Socket.Readable.push (node:internal/streams/readable:228:10)
 at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

Ah, so a failed database migration in the upgrade to 5.57.2 is the culprit! 👏

I brought the site back online quickly by changing the container version for Ghost back to the previous version (5.55.2).

Why did the database migration fail?

The error message from above boils down to this:

Error: alter table `donation_payment_events` add constraint 
`donation_payment_events_member_id_foreign` foreign key (`member_id`)
references `members` (`id`) on delete SET NULL - Referencing column
'member_id' and referenced column 'id' in foreign key constraint 
'donation_payment_events_member_id_foreign' are incompatible.

Adjusting the donation_payment_events.member_id column to be a foreign key of members.id is failing because they are incompatible types. However, as I examined both tables, both were regular varchar(24) columns without anything special attached to them:

mysql> describe members;
+------------------------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------------+---------------+------+-----+---------+-------+
| id | varchar(24) | NO | PRI | NULL | |
| uuid | varchar(36) | YES | UNI | NULL | |
| email | varchar(191) | NO | UNI | NULL | |
| status | varchar(50) | NO | | free | |
| name | varchar(191) | YES | | NULL | |
| expertise | varchar(191) | YES | | NULL | |
| note | varchar(2000) | YES | | NULL | |
| geolocation | varchar(2000) | YES | | NULL | |
| enable_comment_notifications | tinyint(1) | NO | | 1 | |
| email_count | int unsigned | NO | | 0 | |
| email_opened_count | int unsigned | NO | | 0 | |
| email_open_rate | int unsigned | YES | MUL | NULL | |
| last_seen_at | datetime | YES | | NULL | |
| last_commented_at | datetime | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| created_by | varchar(24) | NO | | NULL | |
| updated_at | datetime | YES | | NULL | |
| updated_by | varchar(24) | YES | | NULL | |
+------------------------------+---------------+------+-----+---------+-------+
18 rows in set (0.00 sec)

Going upstream

I went to Ghost’s GitHub repository and opened an issue with as much data as I can find.

One of the first replies mentioned something about database collations. Long story short, collations describe how databases handle sorting and comparing data for different languages. Comparing some languages to other languages can be particularly challenging and this can lead to problems.

I made a switch from MariaDB to MySQL recently for the blog. Could that be related?

More searching

I figured that I wasn’t the first one to stumble into this problem, and sure enough – I wasn’t! There’s a great blog post about a broken migration from MySQL 5 to 8 with Ghost.

In short, it required several steps to fix it:

Stop the Ghost container
Back up the database first (always a good idea)
Do a quick find/replace on the dumped database to change the collations
Drop the ghost database from the database 😱
Import the database back into MySQL
Start Ghost again

Dropping databases always makes me pause, but that’s what backups are for! 😉

How I fixed it

In my case, my MySQL container is called ghostmysql and my Ghost database is ghostdb. Then I made a backup of the database using mysqldump:

sudo docker-compose exec ghostmysql mysqldump \
 -u root -psuper-secret-password ghostdb > backup-ghost-db.sql

Next, I copied the SQL file to another directory just in case I accidentally deleted this backup with an errant command.

cp backup-ghost-db.sql ../

Then I made a copy of the SQL file in the current directory and ran the find and replace on that copy. This changes the collations from the wrong one, utf8mb4_general_ci, to the right one, utf8mb4_0900_ai_ci²:

cp backup-ghost-db.sql backup-ghost-db-new.sql
sed -i 's/utf8mb4_general_ci/utf8mb4_0900_ai_ci/g' \
 backup-ghost-db-new.sql

Now I have the collations right for importing the database back into MySQL. But first, I have to drop the existing database. This is a good time to double check your backups!

sudo docker-compose exec ghostmysql mysql -u root \
 -psuper-secret-password
mysql> DROP DATABASE ghostdb;

Now we can import the modified backup:

cat backup-ghost-db-new.sql | sudo docker-compose exec -T \
 ghostmysql mysql -u root -psuper-secret-password ghostdb

Start all the containers:

sudo docker-compose up -d

Ghost was back online with the older version and everything looked good! I updated my docker-compose.yaml back to use latest for the Ghost version and ran sudo docker-compose up -d once more.

Within seconds, the new container image was in place and the container was running! Both migrations completed in seconds and the blog was back online with the newest version. 🎉

Theta is one of many financial Greeks that measure certain aspects of options contracts in the market. It’s also a letter in the Greek alphabet. ↩︎
The default collation in MySQL 8 is utf8mb4_0900_ai_ci. ↩︎

Launch a watchtower container via podman quadlets

major@mhtx.net (Major Hayden) — Wed, 31 May 2023 00:00:00 +0000

Most of my container workloads run on independent CoreOS cloud instances that I treat like pets. Keeping containers update remains a constant battle, but it’s still easier than running kubernetes.

I wrote about using watchtower in the past to keep containers updated. It’s a simple container that does a few important things:

It monitors (via docker/podman socket) the running containers on the host
It tracks the versions/tags of each container image
It looks for updated versions of the container image in their upstream repositories
Based on a configurable schedule, it pulls a new container image and restarts the container for updates

I encourage you to read more about watchtower on GitHub. There’s plenty you can configure, including update intervals, how updates are handled, and how you can get notifications when an update happens.

My new deployments always need watchtower running. Luckily, we can combine Fedora CoreOS’ initial provisioning system, called ignition, with podman’s new quadlet feature and launch watchtower automatically on the first boot.

Quadlets

So what’s a quadlet?

The blog post explains it well by making containers more declarative via a familiar systemd syntax. Here’s an example .container file from the post:

[Unit]
Description=The sleep container
After=local-fs.target

[Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

You can toss this into $HOME/.config/containers/systemd/mysleep.container for rootless user containers or in /etc/containers/systemd/mysleep.container for a container running as root.

Configure a quadlet on boot

As I mentioned earlier, I want a watchtower container running on my CoreOS nodes at first boot. Let’s start with a fairly basic butane file:

variant: fcos
version: 1.4.0
passwd:
 users:
 - name: major
 groups:
 - wheel
 - sudo
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0
storage:
 files:
 - path: /etc/containers/systemd/watchtower.container
 contents:
 inline: |
 [Unit]
 Description=Watchtower container updater
 Wants=network-online.target
 After=network-online.target

 [Container]
 ContainerName=watchtower
 Image=ghcr.io/containrrr/watchtower:1.5.3@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca
 Environment=WATCHTOWER_CLEANUP=true
 Environment=WATCHTOWER_POLL_INTERVAL=3600
 Volume=/var/run/docker.sock:/var/run/docker.sock
 SecurityLabelDisable=true

 [Install]
 WantedBy=multi-user.target default.target

Let’s break this file down:

I start by adding a user named major that has administrative privileges an an ssh key (this is optional, but I like using my own username rather than core)
The quadlet unit file lands in /etc/containers/systemd/watchtower.container and starts at boot time

The quadlet file has some important configurations:

I added environment variables to clean up outdated container images and check for updates once an hour
The podman socket is mounted inside the watchtower container
Security labels are disabled to allow for communication with the podman socket

[!NOTE] Mounting the podman socket and disabling security labels is not an ideal security approach. However, I’ve found that watchtower’s configuration and automation fits my needs really well and I retreive the image from a trusted source. If this won’t work for you, you can use podman’s built-in auto-update feature instead.

From here, we convert the butane configuration into an ignition configuration. I’m launching this CoreOS node on VULTR, so I’ve named my files accordingly:

$ butane vultr-coreos.butane > vultr-coreos.ign

Let’s go 🚀

I’m using VULTR’s CLI here in Fedora, but you can do the same steps via VULTR’s portal if needed. Just paste in the ignition configuration into the large text box before launch.

# Install vultr-cli in Fedora
sudo dnf install vultr-cli

# Launch the instance
vultr-cli instance create --region dfw --plan vhp-2c-2gb-amd \
 --os 391 --label coreos-dfw-1 --host coreos-dfw-1 \
 --userdata "$(cat vultr-coreos.ign)"

Let’s see how the container is doing:

$ ssh major@COREOS_HOST

Fedora CoreOS 38.20230430.3.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[major@coreos-dfw-1 ~]$ sudo podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0024712c95d ghcr.io/containrrr/watchtower@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca About a minute ago Up About a minute watchtower

[major@coreos-dfw-1 ~]$ sudo podman logs watchtower
time="2023-05-31T14:01:12Z" level=info msg="Watchtower 1.5.3"
time="2023-05-31T14:01:12Z" level=info msg="Using no notifications"
time="2023-05-31T14:01:12Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2023-05-31T14:01:12Z" level=info msg="Scheduling first run: 2023-05-31 15:01:12 +0000 UTC"
time="2023-05-31T14:01:12Z" level=info msg="Note that the first check will be performed in 59 minutes, 59 seconds"

Awesome! 🥳

My system rebooted for an ostree update shortly after provisioning and the container came up automatically both times.

CoreOS as a pet

major@mhtx.net (Major Hayden) — Thu, 25 May 2023 00:00:00 +0000

Anyone working with containers has likely heard of CoreOS by this point Haven’t heard about it? Don’t despair. I’ll catch you up on what you missed.

Fedora CoreOS offers a really fast pathway to running containers on hardware, in virtual machines, or in clouds. It delivers a lightweight operating system with all of the container technology that you need for running simple containers or launching a kubernetes deployment.

But that’s not the best part.

CoreOS really shines due to its immutable OS layer¹. The OS underneath your containers ships as a single unit and it automatically updates itself much like your mobile phone. An update rolls down, CoreOS sets it up as a secondary OS, reboots into that new update, and rolls back to the original one if there were any issues.

Many people use CoreOS as the workhorse underneath kubernetes. Red Hat uses it underneath OpenShift as well. It’s even supported by the super light weight kubernetes distribution k3s.

But can you use CoreOS as a pet type instance that you use and maintain for long periods of time just like any other server? Absolutely!

What’s this pet stuff about?

Whether you like it or not, there’s a cattle versus pets paradigm that took hold in the world of IT at some point. The basic ideas are these:

When you take care of cattle, you take care of them as a group. Losing one or more of them would make you sad, but you know you have many others.
As for pets, you spend a lot of time taking care of them and playing with them. If you lost one, it would be devastating.

A fleet of web servers could be treated like cattle. Keep lots of them online and replace any instances that have issues.

On the other hand, databases or tier zero systems (everyone feels if it they went down) are like pets. You carefully build, maintain, and monitor these.

How does CoreOS fit in?

Many people do use CoreOS as a container hosting platform as part of a bigger system. It works really well for that. But it’s great as a regular cloud server, too.

You can run a single node CoreOS deployment and manage containers via the tools that you know and love. For example, docker-compose works great on CoreOS. I even used it to host my own Mastodon deployment.

You can also load up more user-friendly tools such as portainer to manage containers in a browser.

My development tools are missing!

😱 No vim? This is too minimal! What are we going to do?

Luckily CoreOS comes with toolbox. 🧰

Toolbox gives you the ability to run a utility container on the system with some handy benefits:

Toolbox environments have seamless access to the user’s home directory, the Wayland and X11 sockets, networking (including Avahi), removable devices (like USB sticks), systemd journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc..

This means that the toolbox feels like a second OS on the system and it has all of the elevated privileges that you need to do your work. Simply run toolbox enter, follow the prompts, and you’ll end up with a Fedora toolbox that matches your CoreOS version. Need a different version, such as Fedora Rawhide? Just specify the Fedora release you want on the prompt:

$ toolbox enter --release 39 
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:39 (500MB)? [y/N]: y

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

⬢[major@toolbox ~]$

Look at the toolbox create --help output to see how to create lots of different toolbox containers with different names and releases. If you go overboard and need to delete some toolboxes, just list your toolboxes with toolbox list and follow it up with toolbox rm.

My tool won’t work in the toolbox.

Some applications have issues running inside a container, even one that has elevated privileges on the system. CoreOS offers an option for layering packages on top of the underlying immutable OS.

Simply run rpm-ostree install PACKAGE to layer a package on top of the OS. When rpm-ostree runs, it creates a new layer and sets that layer to be active on the next boot. That means that you need to reboot before you can use the package.

Don’t want to reboot? There’s another option, but I recommend against it if you can avoid it².

You can apply a package layer live on the system without a reboot with the --apply-live flag. Installing a package like mtr would look like this:

$ sudo rpm-ostree install --apply-live mtr

As soon as rpm-ostree finishes its work, mtr should be available on the system for you to use.

How do updates work?

There are two main technologies at work here.

First, zincati checks for updates to your immutable OS tree. It runs on a configurable schedule that you can adjust based on your preferences.

Second, rpm-ostree handles the OS layers and switches between them at boot time. If you’re running off layer A and an update comes down (layer B), that layer is written to the disk and activated on the next boot. Should there be any issues booting up layer B later, rpm-ostree switches the system back to layer A. In these situations, your downtime might be extended a bit due to two reboots. Your system will come back up with the original OS layer activated.

You also get a choice of update streams. Want to live a bit more on the edge? Go for next or testing. You’re on the stable stream by default.

Although I haven’t landed in this situation, it’s possible that the system boots into a new update where you notice a problem that doesn’t affect the boot. You can manually roll back to fix it.

I have more questions.

Your first stop should be the Fedora CoreOS docs. There are also lots of ways to contact the development team and talk with the community.

Love the idea of an immutable OS but you wish you had it for your desktop or laptop? Go check out Fedora Silverblue. 💻

Okay, so it’s mostly immutable. You can edit configuration in /etc and you can layer more packages on top of the base OS layer if you need them. However, CoreOS maintainers discourage adding layered packages if you can avoid it. ↩︎
When you apply some packages and make them available immediately, you may lose track of which ones were applied live and which ones are available on the next reboot. Things can get a bit confusing if you suddenly change your mind about applying a package live or not. ↩︎

Run Xorg applications with podman

major@mhtx.net (Major Hayden) — Sun, 17 Oct 2021 00:00:00 +0000

Containers are a great way to deliver and run all kinds of applications. Although many people build containers for server applications, you can also use them for client applications on your local workstation. This helps when you want to test new applications without disrupting your existing system or when you use an immutable system such as Fedora Silverblue.

Podman takes this further by allowing you to run a client application without root access or daemons. This post covers how to build a container with an Xorg application and run it on a Fedora system.

Building the container

Let’s start with a simple container that contains xeyes. This simple application simply puts a pair of eyes on your screen that follow your mouse movements around the desktop. It has very few dependencies and it’s a great way to test several capabilities on the desktop.

Here’s a very simple container build file:

# xeyes-container
FROM registry.fedoraproject.org/fedora:latest
RUN dnf -y install xeyes
CMD xeyes

Let’s install podman and build the container:

$ sudo dnf -y install podman
$ podman build -t xeyes -f xeyes-container .

Run the container

Now that we have our xeyes container, let’s run it.

$ podman run --rm xeyes
Error: Can't open display:

We’re missing the DISPLAY variable inside the container. Let’s add it:

$ echo $DISPLAY
:0
$ podman run --rm -e DISPLAY xeyes
Error: Can't open display: :0

Well, we have the display variable inside now, but there’s another problem. Inside the container, xeyes can’t make a connection to our X daemon. This socket normally appears in /tmp/.X11-unix, but the container doesn’t have it. Let’s try adding this inside the container:

$ podman run --rm -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix xeyes
Error: Can't open display: :0

Darn! This should be working. Let’s check the system journal:

AVC avc: denied { write } for pid=10817 comm="xeyes" name="X0" dev="tmpfs"
ino=42 scontext=system_u:system_r:container_t:s0:c143,c574
tcontext=system_u:object_r:user_tmp_t:s0 tclass=sock_file permissive=0

Uh oh. SELinux is upset that a container is trying to mess with the X0 socket for our Xorg server that sits in /tmp/.X11-unix. You may be tempted to run setenforce 0, but wait. We can fix this with podman!

Podman allows you to set security options for a particular container with --security-opt. We need to run this container with an SELinux context that allows it to talk to something in tmpfs. Examining the container-selinux project shows that container_runtime_t can work with tmpfs:

type container_runtime_tmp_t alias docker_tmp_t;
files_tmp_file(container_runtime_tmp_t)

Let’s try adding this to our podman command now:

$ podman run --rm -e DISPLAY \
 -v /tmp/.X11-unix:/tmp/.X11-unix \
 --security-opt label=type:container_runtime_t xeyes

I now have a set of eyeballs on my desktop! 👀

Extra credit

The xeyes application is extremely simple, but you can run much more complex applications using this same method. Keep in mind that certain applications might require extra packages inside the container, such as fonts or GTK themes. Jessie Frazelle has a great repository full of containers that she uses regularly and this might give you inspiration to create some of your own! 🤓

Photo credit: Jonny Gios on Unsplash

Rootless container management with docker-compose and podman

major@mhtx.net (Major Hayden) — Fri, 09 Jul 2021 00:00:00 +0000

Everyone has an opinion for the best way to manage containers, and there are many contenders depending on how much complexity you can handle and how much automation you require. One of my favorite ways to manage containers is docker-compose.

Overview of docker-compose

docker-compose uses a simple YAML syntax to explain what your desired end state should look like. The compose specification covers all of the relevant configurations for containers, volumes, networks, and more. After each change, docker-compose compares your configuration to the running containers and makes all of the required changes.

This provides some advantages over using docker run ... or podman run ... since you can put the YAML into version control and track your configuration changes all in one place. I was tracking the configuration in shell scripts that ran docker with lots of parameters and that became difficult to manage.

What about Podman?

Podman is a tool for managing containers, much like Docker, but it has some distinct advantages:

No daemons are needed
You can run containers as your user, or as root
The commands and arguments are nearly identical to docker (no swarm support)
Podman 3 added a complete Docker-compatible API

This last part, the Docker-compatible API is quite interesting and this allows docker-compose to work with podman as well as it does with docker.

Let’s try it out!

Getting everything ready

Start with a working Fedora 34 system and install some packages:

💣 HEADS UP: The podman-docker package brings in podman, an alias for the docker command that actually runs podman, and the docker-compatible API via a socket. If you want to run podman and docker side by side on the same machine, install podman instead of podman-docker here. If you had docker installed already, you may need to remove it with dnf remove docker-ce docker-ce-cli.

dnf install docker-compose podman-docker

We’re going to do something different here. Intead of starting the podman socket or docker daemon as root, we’re going to start the podman socket as a regular user. Switch to a regular user and start the socket:

$ systemctl enable --now --user podman.socket
Created symlink /home/major/.config/systemd/user/sockets.target.wants/podman.socket → /usr/lib/systemd/user/podman.socket.

But wait, where’s the socket?

$ ls -al $XDG_RUNTIME_DIR/podman/podman.sock
srw-rw----. 1 major major 0 Jul 9 16:49 /run/user/1000/podman/podman.sock

That’s a podman socket running as my user and exposing a docker-compatible API. 🎉

Time for docker-compose

Now it’s time to use docker-compose with podman as a regular user and run a container as our regular user.

We can use librespeed for this example, and the LinuxServer librespeed container is a great way to deploy it. It’s a self-hosted speed test application that works well with desktops and mobile devices.

First, we begin with the suggested docker-compose configuration:

---
version: "2.1"
services:
 librespeed:
 image: ghcr.io/linuxserver/librespeed
 container_name: librespeed
 environment:
 - PUID=1000
 - PGID=1000
 - TZ=Etc/UTC
 volumes:
 - librespeed:/config
 ports:
 - 8080:80
 restart: unless-stopped

volumes:
 librespeed: {}

Save that as docker-compose.yml in your current directory.

Keep in mind that docker-compose is expecting to find our docker socket in /var/run/docker.sock, but we’re running the podman socket as our regular user. Let’s export the DOCKER_HOST variable and run docker-compose to bring up our new container:

$ export DOCKER_HOST="unix:$XDG_RUNTIME_DIR/podman/podman.sock"
$ docker-compose up -d
Pulling librespeed (ghcr.io/linuxserver/librespeed:)...
10f45b17b9ab: Download complete
f23b92877416: Download complete
a5bf9c523af4: Download complete
00fe9b963179: Download complete
bfafa0ba1dc9: Download complete
c583b34264f1: Download complete
9d26cce56b8d: Download complete
70de87880afd: Download complete
0ad6c2578069: Download complete
a8792749de3b: Download complete
2d31530d2d8b: Download complete
Creating librespeed ... done
$ docker-compose ps
 Name Command State Ports
--------------------------------------------
librespeed /init Up () :8080->80/tcp

The container is up and running as our user. Let’s check the nginx process inside the container to be sure:

$ ps -xu |grep "nginx: master"
major 3805 0.0 0.4 5860 4692 ? Ss 16:53 0:00 nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf

Sweet! 🥳

Time for a speed test

If we’ve come this far, we might as well test our internet speed to ensure the container works!

Remember that we used port 8080 as a replacement for 80 in our docker-compose file to avoid issues with regular users being denied access to create a listener on ports under 1024.

Let’s see how fast my connection is today:

Photo credit: Michael D Beckwith on Unsplash

Get faster GitLab runners with a ramdisk

major@mhtx.net (Major Hayden) — Fri, 16 Aug 2019 00:00:00 +0000

When you build tons of kernels every day like my team does, you look for speed improvements anywhere you can. Caching repositories, artifacts, and compiled objects makes kernel builds faster and it reduces infrastructure costs.

Need for speed

We use GitLab CI in plenty of places, and that means we have a lot of gitlab-runner configurations for OpenShift (using the kubernetes executor) and AWS (using the docker-machine executor). The runner’s built-in caching makes it easy to upload and download cached items from object storage repositories like Google Cloud Storage or Amazon S3.

However, there’s an often overlooked feature hiding in the configuration for the docker executor that provides a great performance boost: mounting tmpfs inside your container. Not familiar with tmpfs? Arch Linux has a great wiki page for tmpfs and James Coyle has a well-written blog post about what makes it unique from the older ramfs.

RAM is much faster than your average cloud provider’s block storage. It also has incredibly low latency relative to most storage media. There’s a great interactive latency page that allows you to use a slider to travel back in time to 1990 and compare all kinds of storage performance numbers. (It’s really fun! Go drag the slider and be amazed.)

Better yet, many cloud providers give you lots of RAM per CPU on their instances, so if your work isn’t terribly memory intensive, you can use a lot of this RAM for faster storage.

Enabling tmpfs in Docker containers

⚠️ Beware of the dangers of tmpfs before adjusting your runner configuration! See the warnings at the end of this post.

This configuration is buried in the middle of the docker executor documentation. You will need to add some extra configuration to your [runners.docker] section to make it work:

[runners.docker]
 [runners.docker.tmpfs]
 "/ramdisk" = "rw,noexec"

This configuration mounts a tmpfs volume underneath /ramdisk inside the container. By default, this directory will be mounted with noexec, but if you need to execute scripts from that directory, change noexec to exec:

[runners.docker]
 [runners.docker.tmpfs]
 "/ramdisk" = "rw,exec"

In our case, compiling kernels requires executing scripts, so we use exec for our tmpfs mounts.

You must be specific for exec! As an example, this tmpfs volume will be mounted with noexec since that is the default:

[runners.docker]
 [runners.docker.tmpfs]
 "/ramdisk" = "rw"

Extra speed

For even more speed, we moved the objects generated by ccache to the ramdisk. The seek times are much lower and this allows ccache to look for its cached objects much more quickly.

Git repositories are also great things to stash on tmpfs. Big kernel repositories are usually 1.5GB to 2GB in size with tons of files. Checkouts are really fast when they’re done in tmpfs.

Dangers are lurking

⚠️ As mentioned earlier, beware of the dangers of tmpfs.

All of the containers on the machine will share the same amount of RAM for their tmpfs volumes. Be sure to account for how much each container will use and how many containers could be present on the same machine.
Be aware of how much memory your tests will use when they run. In our case, kernel compiles can consume 2-4GB of RAM, depending on configuration, so we try our best to leave some memory free.
These volumes also have no limits on how much data can go into the volume. However, if you put too much data into the tmpfs volume and your system runs critically low on available RAM, you could see a huge drop in performance, system instability, or even a crash. 🔥

buildah error: vfs driver does not support overlay.mountopt options

major@mhtx.net (Major Hayden) — Tue, 13 Aug 2019 00:00:00 +0000

Buildah and podman make a great pair for building, managing and running containers on a Linux system. You can even use them with GitLab CI with a few small adjustments, namely the switch from the overlayfs to vfs storage driver.

I have some regularly scheduled GitLab CI jobs that attempt to build fresh containers each morning and I use these to get the latest packages and find out early when something is broken in the build process. A failed build appeared in my inbox earlier this week with the following error:

+ buildah bud -f builds/builder-fedora30 -t builder-fedora30 .
vfs driver does not support overlay.mountopt options

My container build script¹ is fairly basic, but it does include a change to use the vfs storage driver:

# Use vfs with buildah. Docker offers overlayfs as a default, but buildah
# cannot stack overlayfs on top of another overlayfs filesystem.
export STORAGE_DRIVER=vfs

The script doesn’t change any mount options during the build process. A quick glance at the /etc/containers/storage.conf revealed a possible problem:

[storage.options]
# Storage options to be passed to underlying storage drivers

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

These mount options make sense when used with an overlayfs filesystem, but they are not used with vfs. I commented out the mountopt option, saved the file, and ran a test build locally. Success!

Fixing the build script involved a small change to the storage.conf just before building the container:

# Use vfs with buildah. Docker offers overlayfs as a default, but buildah
# cannot stack overlayfs on top of another overlayfs filesystem.
export STORAGE_DRIVER=vfs

# Newer versions of podman/buildah try to set overlayfs mount options when
# using the vfs driver, and this causes errors.
sed -i '/^mountopt =.*/d' /etc/containers/storage.conf

My containers are happily building again in GitLab.

The original build script is no longer available, but the remainder of the repository still exists. ↩︎

Build containers in GitLab CI with buildah

major@mhtx.net (Major Hayden) — Fri, 24 May 2019 00:00:00 +0000

My team at Red Hat depends heavily on GitLab CI and we build containers often to run all kinds of tests. Fortunately, GitLab offers up CI to build containers and a container registry in every repository to hold the containers we build.

This is really handy because it keeps everything together in one place: your container build scripts, your container build infrastructure, and the registry that holds your containers. Better yet, you can put multiple types of containers underneath a single git repository if you need to build containers based on different Linux distributions.

Building with Docker in GitLab CI

By default, GitLab offers up a Docker builder that works just fine. The CI system clones your repository, builds your containers and pushes them wherever you want. There’s even a simple CI YAML file that does everything end-to-end for you.

However, I have two issues with the Docker builder:

Larger images: The Docker image layering is handy, but the images end up being a bit larger, especially if you don’t do a little cleanup in each stage.
Additional service: It requires an additional service inside the CI runner for the dind (“Docker in Docker”) builder. This has caused some CI delays for me several times.

Building with buildah in GitLab CI

On my local workstation, I use podman and buildah all the time to build, run, and test containers. These tools are handy because I don’t need to remember to start the Docker daemon each time I want to mess with a container. I also don’t need sudo.

All of my containers are stored beneath my home directory. That’s good for keeping disk space in check, but it’s especially helpful on shared servers since each user has their own unique storage. My container pulls and builds won’t disrupt anyone else’s work on the server and their work won’t disrupt mine.

Finally, buildah offers some nice options out of the box. First, when you build a container with buildah bud, you end up with only three layers by default:

Original OS layer (example: fedora:30)
Everything you added on top of the OS layer
Tiny bit of metadata

This is incredibly helpful if you use package managers like dnf, apt, and yum that download a bunch of metadata before installing packages. You would normally have to clear the metadata carefully for the package manager so that your container wouldn’t grow in size. Buildah takes care of that by squashing all the stuff you add into one layer.

Of course, if you want to be more aggressive, buildah offers the --squash option which squashes the whole image down into one layer. This can be helpful if disk space is at a premium and you change the layers often.

Getting started

I have a repository called os-containers in GitLab that maintains fully updated containers for Fedora 29 and 30. The .gitlab-ci.yml file calls build.sh for two containers: fedora29 and fedora30. Open the build.sh file and follow along here:

# Use vfs with buildah. Docker offers overlayfs as a default, but buildah
# cannot stack overlayfs on top of another overlayfs filesystem.
export STORAGE_DRIVER=vfs

First off, we need to tell buildah to use the vfs storage driver. Docker uses overlayfs by default and stacking overlay filesystems will definitely lead to problems. Buildah won’t let you try it.

# Write all image metadata in the docker format, not the standard OCI format.
# Newer versions of docker can handle the OCI format, but older versions, like
# the one shipped with Fedora 30, cannot handle the format.
export BUILDAH_FORMAT=docker

By default, buildah uses the oci container format. This sometimes causes issues with older versions of Docker that don’t understand how to parse that type of metadata. By setting the format to docker, we’re using a format that almost all container runtimes can understand.

# Log into GitLab's container repository.
export REGISTRY_AUTH_FILE=${HOME}/auth.json
echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY

Here we set a path for the auth.json that contains the credentials for talking to the container repository. We also use buildah to authenticate to GitLab’s built-in container repository. GitLab automatically exports these variables for us (and hides them in the job output), so we can use them here.

buildah bud -f builds/${IMAGE_NAME} -t ${IMAGE_NAME} .

We’re now building the container and storing it temporarily as the bare image name, such as fedora30. This is roughly equivalent to docker build.

CONTAINER_ID=$(buildah from ${IMAGE_NAME})
buildah commit --squash $CONTAINER_ID $FQ_IMAGE_NAME

Now we are making a reference to our container with buildah from and using that reference to squash that container down into a single layer. This keeps the container as small as possible.

The commit step also tags the resulting image using our fully qualified image name (in this case, it’s registry.gitlab.com/majorhayden/os-containers/fedora30:latest)

buildah push ${FQ_IMAGE_NAME}

This is the same as docker push. There’s not much special to see here.

Maintaining containers

GitLab allows you to take things to the next level with CI schedules. In my repository, there is a schedule to build my containers once a day to catch the latest updates. I use these containers a lot and they need to be up to date before I can run tests.

If the container build fails for some reason, GitLab will send me an email to let me know.

Podman on Major Hayden

Getting podman quadlets talking to each other

Setting up some quadlets

Testing communication

Fixing communication

Testing communication again

Extra credit

Automatic container updates with Podman quadlets

Setting up a basic quadlet

Enabling automatic updates

Customizing update behavior

Monitoring updates

Further reading

How I learned to stop worrying and love the CoreOS

Updates

Toolbox

Layering

Declarative provisioning

Pets and cattle

More questions?

Quadlets might make me finally stop using docker-compose

What’s a quadlet?

So why not stick with docker-compose?

Quaint quadlets quickly!

Launch the quadlets

Managing containers

Fixing a ghost database migration failure

Why is the site down?

Why did the database migration fail?

Going upstream

More searching

How I fixed it

Launch a watchtower container via podman quadlets

Quadlets

Configure a quadlet on boot

Let’s go 🚀

CoreOS as a pet

What’s this pet stuff about?

How does CoreOS fit in?

My development tools are missing!

My tool won’t work in the toolbox.

How do updates work?

I have more questions.

Run Xorg applications with podman

Building the container

Run the container

Extra credit

Rootless container management with docker-compose and podman

Overview of docker-compose

What about Podman?

Getting everything ready

Time for docker-compose

Time for a speed test

Get faster GitLab runners with a ramdisk

Need for speed

Enabling tmpfs in Docker containers

Extra speed

Dangers are lurking

buildah error: vfs driver does not support overlay.mountopt options

Build containers in GitLab CI with buildah

Building with Docker in GitLab CI

Building with buildah in GitLab CI

Getting started

Maintaining containers