Major Hayden

Jellyfin fatal player error

major@mhtx.net (Major Hayden) — Tue, 02 Jul 2024 00:00:00 +0000

Plex has been a mainstay for serving up media at home but it seems to have changed lately towards a more and more commercial offering. A friend recommended Jellyfin and I deployed it on my Synology NAS in a Docker container.

I did a few quick tests in a web browser and everything looked good. But then my Jellyfin android app told me:

Playback failed due to a fatal player error

Everything looked fine in the browser, so it was time to dig in.

Checking the logs #

I opened up an ssh connection on the Synology to check the logs and found something unhelpful:

Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 1

Running a few searches led me down rabbit holes to plenty of GitHub issues. None of them fixed the issue.

Checking the browser again #

I went through a few different videos from the Synology and played each. They all looked fine in Firefox until I reached one that seemed to stutter. The frame rate looked as if at least half of the frames were bring dropped.

That particular video was in 4K with a high bit rate. Back on the synology, the CPU usage was through the roof.

I configured graphics acceleration when I deployed Jellyfin. Perhaps it wasn’t working?

Jellyfin deployment #

I deployed Jellyfin using the upstream guides with docker-compose:

jellyfin:
 image: docker.io/jellyfin/jellyfin:latest
 container_name: jellyfin
 user: 1026:100
 network_mode: "host"
 devices:
 - /dev/dri
 volumes:
 # removed
 restart: "unless-stopped"

One of the GitHub issues I stumbled upon suggested being specific about the video devices that are mounted inside the container.

$ ls -al /dev/dri
total 0
drwxr-xr-x 2 root root 80 Jun 10 20:23 .
drwxr-xr-x 12 root root 14140 Jun 10 20:24 ..
crw------- 1 root root 226, 0 Jun 10 20:24 card0
crw-rw---- 1 root videodriver 226, 128 Jun 10 20:24 renderD128

I adjusted the deployment in docker-compose.yaml and tried again:

jellyfin:
 image: docker.io/jellyfin/jellyfin:latest
 container_name: jellyfin
 user: 1026:100
 network_mode: "host"
 devices:
 - /dev/dri/renderD128:/dev/dri/renderD128
 - /dev/dri/card0:/dev/dri/card0
 volumes:
 # removed
 restart: "unless-stopped"

I redeployed jellyfin:

$ docker-compose up -d jellyfin

The Android app still had the fatal player error.

Users and groups #

Most of my Synology containers use the uid/gid pair of 1026:100 so allow them to read and write to my storage volume. The /dev/dri/renderD128 is owned by the videodriver group:

$ grep videodriver /etc/group
videodriver::937:PlexMediaServer

This likely came from a time when I installed Plex on Synology via one of the Synology applications rather than from a container. (I’m not sure, but that’s my guess.)

I added that group to the container:

jellyfin:
 image: docker.io/jellyfin/jellyfin:latest
 container_name: jellyfin
 user: 1026:100
 network_mode: "host"
 group_add:
 - "937"
 devices:
 - /dev/dri/renderD128:/dev/dri/renderD128
 - /dev/dri/card0:/dev/dri/card0
 volumes:
 # removed
 restart: "unless-stopped"

After redeploying the container, the Android app worked just fine! Also, the video stuttering disappeared when viewing the 4K video from the browser. 🎉

Redirect local ports with firewalld

major@mhtx.net (Major Hayden) — Fri, 28 Jun 2024 00:00:00 +0000

Linux networking and firewalls give us plenty of options for redirecting traffic from one port to another. We can allow people outside our home to reach a web server we run in our internal network. That’s called destination NAT, ot DNAT.

You can also redirect traffic to different ports on the same host. For example, if you have a daemon listening on port 3000, but you want people to reach that service on port 80, you can redirect traffic from 80 to 3000 on the same host (without network address translation).

But how do we do this with firewalld? 🤔

Old-school iptables methods #

Let’s say you have a service running on port 3000 and you want to expose it to other computers on your same network as port 80. With iptables, you would typically start by enabling IP forwarding:

sudo sysctl -w net.ipv4.ip_forward=1

Add two iptables rules to handle packets coming in from the outside as well as any locally generated packets:

# Handle locally-generated packets on the same machine.
sudo iptables -t nat -A PREROUTING -s 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to 3000`

# Handle packets coming from outside the current machine.
sudo iptables -t nat -A OUTPUT -s 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to 3000`

There’s a weird situation that happens on certain machines with certain network configurations where packets are not properly routed when they are destined for the local network adapter. To fix that, set one more sysctl configuration:

sudo sysctl -w net.ipv4.conf.all.route_localnet=1

Remember to make these sysctl configurations permanent:

sudo mkdir /etc/sysctl.conf.d/
echo "net.ipv4.ip_forward=1" | sudo tee >> /etc/sysctl.conf.d/redirect.conf
echo "net.ipv4.conf.all.route_localnet" | sudo tee >> /etc/sysctl.conf.d/redirect.conf

Why consider firewalld? #

I like firewalld because I can manage lots of settings for different firewall zones and allow access from one zone to another. It also allows me to put certain interfaces in trusted zones so they automatically get more access.

Another nice aspect about firewalld is that it supports iptables and nftables backends. You don’t have to think about the differences between the backends. All of that is taken care of for you.

Port redirections in firewalld #

Let’s start by checking our default firewalld zone:

$ sudo firewall-cmd --list-all
FedoraServer (default, active)
 target: default
 ingress-priority: 0
 egress-priority: 0
 icmp-block-inversion: no
 interfaces: bond0 eno1 eno2
 sources: 
 services: dhcpv6-client http https
 ports: 51820/udp
 protocols: 
 forward: yes
 masquerade: yes
 forward-ports:
 source-ports: 
 icmp-blocks: 
 rich rules:

This output shows that my external network interfaces are attached to the zone and forwarding is already on in my case. If you see forward: no here, just run this command:

$ sudo firewall-cmd --add-forward
success

Now firewalld will manage your forwarding sysctl variables for you on these interfaces. That’s handy. 😉

Next, let’s get the redirect working. We want to take external packets on port 80 and send them to 3000: on the local machine.

$ sudo firewall-cmd \
 --add-forward-port=port=80:proto=tcp:toport=3000:toaddr=127.0.0.1
success

In this command, we told firewalld to take 80/tcp from the outside and send it to port 3000 on the local host (127.0.0.1). Let’s double check our current configuration:

$ sudo firewall-cmd --list-all
FedoraServer (default, active)
 target: default
 ingress-priority: 0
 egress-priority: 0
 icmp-block-inversion: no
 interfaces: bond0 eno1 eno2
 sources: 
 services: dhcpv6-client http https
 ports: 51820/udp
 protocols: 
 forward: yes
 masquerade: yes
 forward-ports: 
	port=80:proto=tcp:toport=3000:toaddr=127.0.0.1
 source-ports: 
 icmp-blocks: 
 rich rules:

Test a connection to port 80 with curl and it should redirect to the service on port 3000.

🚨 If everything works, remember to save the firewalld configuration:

$ sudo firewall-cmd --runtime-to-permanent
success

Extra credit #

We can inspect the nftables rules to see the firewall rules that firewalld set for us. The Arch Linux nftables wiki page is superb for looking up those commands.

If we dump the current ruleset, we see the rule we created in firewalld:

$ sudo nft list ruleset
---SNIP---
chain nat_PRE_FedoraServer_allow {
 meta nfproto ipv4 tcp dport 80 dnat ip to 127.0.0.1:3000
}
---SNIP---

amazon-ec2-utils in Fedora

major@mhtx.net (Major Hayden) — Wed, 08 May 2024 00:00:00 +0000

We’ve all been in that situation where we see a device in Linux and wonder which physical device it corresponds to. I remember when I built my first NAS and received an alert that a drive had failed. It took me a while to figure out which physical drive actually needed to be replaced.

This happens with network devices, too, and I wrote a post about systemd’s predictable network device names back in 2015.

Cloud instances often make it even more confusing because storage devices are fully virtualized and show up differently depending on the cloud provider. I recently packaged amazon-ec2-utils in Fedora to make this a little easier on AWS.

The problem #

I just built a test instance of Fedora 40 in AWS and the AWS API shows the block device mappings like this:

$ aws ec2 describe-instances \
 --instance-ids i-0687448a184ab0a9e | \
 jq '.Reservations[0].Instances[0].BlockDeviceMappings'
[
 {
 "DeviceName": "/dev/sda1",
 "Ebs": {
 "AttachTime": "2024-05-08T15:24:03+00:00",
 "DeleteOnTermination": true,
 "Status": "attached",
 "VolumeId": "vol-0832569729b6c5ea6"
 }
 }
]

However, if I check these devices inside the instance itself, I get something totally different:

[fedora@f40 ~]$ sudo fdisk -l
Disk /dev/nvme0n1: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Amazon Elastic Block Store 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9FB58ED7-7581-4469-BEB7-64F069151EAF

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 206847 204800 100M EFI System
/dev/nvme0n1p2 206848 2254847 2048000 1000M Linux extended boot
/dev/nvme0n1p3 2254848 20971484 18716637 8.9G Linux root (ARM-64)


Disk /dev/zram0: 1.78 GiB, 1909456896 bytes, 466176 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

[fedora@f40 ~]$ ls -al /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory

One disk isn’t so bad, but what if we add more storage? The API tells me one thing:

> aws ec2 describe-instances --instance-ids i-0687448a184ab0a9e | jq '.Reservations[0].Instances[0].BlockDeviceMappings'
[
 {
 "DeviceName": "/dev/sda1",
 "Ebs": {
 "AttachTime": "2024-05-08T15:24:03+00:00",
 "DeleteOnTermination": true,
 "Status": "attached",
 "VolumeId": "vol-0832569729b6c5ea6"
 }
 },
 {
 "DeviceName": "/dev/sde",
 "Ebs": {
 "AttachTime": "2024-05-08T15:38:29.754000+00:00",
 "DeleteOnTermination": false,
 "Status": "attached",
 "VolumeId": "vol-0a7ba05c5270d7aa3",
 "VolumeOwnerId": "xxx"
 }
 }
]

But then the instance tells me something else entirely:

[fedora@f40 ~]$ sudo fdisk -l
Disk /dev/nvme0n1: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Amazon Elastic Block Store 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9FB58ED7-7581-4469-BEB7-64F069151EAF

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 206847 204800 100M EFI System
/dev/nvme0n1p2 206848 2254847 2048000 1000M Linux extended boot
/dev/nvme0n1p3 2254848 20971484 18716637 8.9G Linux root (ARM-64)


Disk /dev/zram0: 1.78 GiB, 1909456896 bytes, 466176 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/nvme1n1: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Amazon Elastic Block Store 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

udev rules to the rescue #

The amazon-ec2-utils package provides some helpful udev rules and scripts to make it easier to identify these devices. This package is on the way to Fedora as I write this post, but it hasn’t reached the stable repos yet. Once it does, you should be able to install it:

$ sudo dnf install amazon-ec2-utils

In the meantime, you can download the latest build and install it on your instance:

$ sudo dnf install /usr/bin/koji
$ koji download-build amazon-ec2-utils-2.2.0-2.fc40 
Downloading [1/2]: amazon-ec2-utils-2.2.0-2.fc40.src.rpm
[====================================] 100% 24.01 KiB / 24.01 KiB
Downloading [2/2]: amazon-ec2-utils-2.2.0-2.fc40.noarch.rpm
[====================================] 100% 20.53 KiB / 20.53 KiB
$ sudo dnf install amazon-ec2-utils-2.2.0-2.fc40.noarch.rpm

The cleanest method to get these new udev rules working is to reboot, but if you’re in a hurry, there’s an option to reload these rules without a reboot:

$ sudo udevadm control --reload-rules
$ sudo udevadm trigger

What do we have in /dev/ now?

[fedora@f40 ~]$ ls -al /dev/sd*
lrwxrwxrwx. 1 root root 7 May 8 15:44 /dev/sda1 -> nvme0n1
lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda11 -> nvme0n1p1
lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda12 -> nvme0n1p2
lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda13 -> nvme0n1p3
lrwxrwxrwx. 1 root root 7 May 8 15:44 /dev/sde -> nvme1n1

We can put a filesystem down on the new device using the same name as the API presents:

$ sudo dnf install /usr/sbin/mkfs.btrfs
$ sudo mkfs.btrfs /dev/sde
btrfs-progs v6.8.1
See https://btrfs.readthedocs.io for more information.

Performing full device TRIM /dev/sde (10.00GiB) ...
NOTE: several default settings have changed in version 5.15, please make sure
 this does not affect your deployments:
 - DUP for metadata (-m dup)
 - enabled no-holes (-O no-holes)
 - enabled free-space-tree (-R free-space-tree)

Label: (null)
UUID: c2fb9e33-3bf6-4b5b-aa80-44e315f499de
Node size: 16384
Sector size: 4096	(CPU page size: 4096)
Filesystem size: 10.00GiB
Block group profiles:
 Data: single 8.00MiB
 Metadata: DUP 256.00MiB
 System: DUP 8.00MiB
SSD detected: yes
Zoned device: no
Features: extref, skinny-metadata, no-holes, free-space-tree
Checksum: crc32c
Number of devices: 1
Devices:
 ID SIZE PATH 
 1 10.00GiB /dev/sde

Being able to know these device names during the instance launch or during storage operations makes it much easier to write automation for these devices. There’s no guess work required to translate the device that an instance shows you to what you see via the API.

Fix big cursors in Java applications in Wayland

major@mhtx.net (Major Hayden) — Fri, 26 Apr 2024 00:00:00 +0000

Scroll through the list of Wayland posts posts on the blog and you’ll see that I’ve solved plenty of weird problems with Wayland and the Sway compositor. Most are pretty easy to fix but some are a bit trickier.

Java applications are notoriously unpredictable and Wayland takes unpredictability to the next level. One particular application on my desktop always seems to start with massive cursors.

This post is about how I fixed and then discovered something interesting along the way.

Fixing big cursors #

I recently moved some investment and trading accounts from TD Ameritrade to Tastytrade. Both offer Java applications that make trading easier, but Tastytrade’s application always started with massive cursors.

To make matters worse, sometimes the cursor looked lined up on the screen but then the click landed on the wrong buttons in the application! Errors are annoying. Errors that cost you money and time must be fixed. 😜

Some web searches eventually led me to Arch Linux’s excellent Wayland wiki page. None of the adjustments or environment variables there had any effect on my cursors.

I eventually landed on a page that suggested setting XCURSOR_SIZE. I don’t remember ever setting that, but it was being set by something:

$ echo $XCURSOR_SIZE
24

One of the suggestions was to decrease it, so I decided to give 20 a try. That was too big, but 16 was perfect and it matched all of my other applications:

$ export XCURSOR_SIZE=20
# /opt/tastytrade/bin/tastytrade

That works fine when I start my application via the terminal, but how do I set it for the application when I start it from ulauncher in sway? 🤔

Desktop file #

The Tastytade RPM comes with a .desktop file for launching the application. I copied that over to my local applications directory:

cp /opt/tastytrade/lib/tastytrade-tastytrade.desktop \
 ~/.local/share/applications/

Then I opened the copied ~/.local/share/applications/tastytrade-tastytrade.desktop file in a text editor:

[Desktop Entry]
Name=tastytrade
Comment=tastytrade
Exec=/opt/tastytrade/bin/tastytrade
Icon=/opt/tastytrade/lib/tastytrade.png
Terminal=false
Type=Application
Categories=tastyworks
MimeType=

I changed the Exec line to be:

Exec=env XCURSOR_SIZE=16 /opt/tastytrade/bin/tastytrade

I launched the application again after making that change, but the cursors were still huge! There has to be another way. 🤔

systemd does everything 😆 #

After more searching and digging, I discovered that systemd has a capability to set environment variables for user sessions:

Configuration files in the environment.d/ directories contain lists of environment variable assignments passed to services started by the systemd user instance. systemd-environment-d-generator(8) parses them and updates the environment exported by the systemd user instance. See below for an discussion of which processes inherit those variables.

It is recommended to use numerical prefixes for file names to simplify ordering.

For backwards compatibility, a symlink to /etc/environment is installed, so this file is also parsed.

Let’s give that a try:

$ mkdir -p ~/.config/environment.d/
$ vim ~/.config/environment.d/wayland.conf

In the file, I added one line with a comment (because you will soon forget why you added it 😄):

# Fix big cursors in Java apps in Wayland
XCURSOR_SIZE=16

After a reboot, I launched my Java application and boom – the cursors were perfect! 🎉

I went back and cleaned up some other hacks I had applied and added them to that wayland.conf file:

# This was important at some point but I'm afraid to remove it.
# Note to self: make detailed comments when adding lines here.
SDL_VIDEODRIVER=wayland
QT_QPA_PLATFORM=wayland

# Reduce window decorations for VLC
QT_WAYLAND_DISABLE_WINDOWDECORATION="1"

# Fix weird window handling when Java apps do certain pop-ups
_JAVA_AWT_WM_NONREPARENTING=1

# Ensure Firefox is using Wayland code (not needed any more)
MOZ_ENABLE_WAYLAND=1

# Disable HiDPI
GDK_SCALE=1

# Fix big cursors in Java apps in Wayland
XCURSOR_SIZE=16

I’m told there are some caveats with this solution, especially if your Wayland desktop doesn’t use systemd to start. This is working for me with GDM launching Sway on Fedora 40.

cloud-init and dhcpcd

major@mhtx.net (Major Hayden) — Thu, 18 Apr 2024 00:00:00 +0000

We’re all familiar with the trusty old dhclient on our Linux systems, but it went end-of-life in 2022:

NOTE: This software is now End-Of-Life. 4.4.3 is the final release
planned. We will continue to keep the public issue tracker and user
mailing list open.

You should read this file carefully before trying to install or use
the ISC DHCP Distribution.

Most Linux distributions use dhclient along with cloud-init for the initial dhcp request during the first part of cloud-init’s work. I set off to switch Fedora’s cloud-init package to dhcpcd instead.

What’s new with dhcpcd? #

There are some nice things about dhcpcd that you can find in the GitHub repository:

Very small footprint with almost no dependencies on Fedora
It can do DHCP and DHCPv6
It can also be a ZeroConf client

The project had its last release back in December 2023 and had commits as recently as this week.

But I use NetworkManager #

That’s great! A switch from dhclient to dhcpcd for cloud-init won’t affect you.

When cloud-init starts, it does an initial dhcp request to get just enough networking to reach the cloud’s metadata service. This service provides all kinds of information for cloud-init, including network setup instructions and initial scripts to run.

NetworkManager doesn’t start taking action until cloud-init has written the network configuration to the system.

But I use systemd-networkd #

Same as with NetworkManager, this change applies to the very early boot and you won’t notice a different when deploying new cloud systems.

How can I get it right now? #

If you’re using a recent build of Fedora rawhide (the unstable release under development), you likely have it right now on your cloud instance. Just run journalctl --boot, search for dhcpcd, and you should see these lines:

cloud-init[725]: Cloud-init v. 24.1.4 running 'init-local' at Wed, 17 Apr 2024 14:39:36 +0000. Up 6.13 seconds.
dhcpcd[727]: dhcpcd-10.0.6 starting
kernel: 8021q: 802.1Q VLAN Support v1.8
dhcpcd[730]: DUID 00:01:00:01:2d:b2:9b:a9:06:eb:18:e7:22:dd
dhcpcd[730]: eth0: IAID 18:e7:22:dd
dhcpcd[730]: eth0: soliciting a DHCP lease
dhcpcd[730]: eth0: offered 172.31.26.195 from 172.31.16.1
dhcpcd[730]: eth0: leased 172.31.26.195 for 3600 seconds
avahi-daemon[706]: Joining mDNS multicast group on interface eth0.IPv4 with address 172.31.26.195.
avahi-daemon[706]: New relevant interface eth0.IPv4 for mDNS.
avahi-daemon[706]: Registering new address record for 172.31.26.195 on eth0.IPv4.
dhcpcd[730]: eth0: adding route to 172.31.16.0/20
dhcpcd[730]: eth0: adding default route via 172.31.16.1
dhcpcd[730]: control command: /usr/sbin/dhcpcd --dumplease --ipv4only eth0

There’s also an update pending for Fedora 40, but it’s currently held up by the beta freeze. That should appear as an update as soon as Fedora 40 is released.

Keep in mind that if you have a system deployed already, cloud-init won’t need to run again. Updating to Fedora 40 will update your cloud-init and pull in dhcpcd, but it won’t need to run again since your configuration is already set.

Texas Linux Fest 2024 recap 🤠

major@mhtx.net (Major Hayden) — Tue, 16 Apr 2024 00:00:00 +0000

The 2024 Texas Linux Festival just ended last weekend and it was a fun event as always. It’s one my favorite events to attend because it’s really casual. You have plenty of opportunities to see old friends, meet new people, and learn a few things along the way.

I was fortunate enough to have two talks accepted for this year’s event. One was focused on containers while the other was a (very belated) addition to my impostor syndrome talk from 2015.

This was also my first time building slides with reveal-md, a “batteries included” package for making reveal.js slides. Nothing broke too badly and that was a relief.

Containers talk #

I’ve wanted to share more of what I’ve done with CoreOS in low-budget container deployments and this seemed like a good time to share it with the world out loud. My talk, Automated container updates with GitHub and CoreOS, walked the audience through how to deploy containers on CoreOS, keep them updated, and update the container image source.

My goal was to keep it as low on budget as possible. Much of it was centered around a stack of caddy, librespeed, and docker-compose. All of it was kept up to date with watchtower.

My custom Caddy container needed support for Porkbun’s DNS API and I used GitHub Actions to build that container and serve it to the internet using GitHub’s package hosting. This also gave me the opportunity to share how awesome Porkbun is for registering domains, including their customized pig artwork for every TLD imaginable. 🐷

We had a great discussion afterwards about how CoreOS does indeed live on as Fedora CoreOS.

Tech career talk #

This talk made me nervous because it had a lot of slides to cover, but I also wanted to leave plenty of time for questions. Five tips for a thriving technology career built upon my old impostor syndrome talk by sharing some of the things I’ve learned over the year that helped me succeed in my career.

I managed to end early with time for questions, and boy did the audience have questions! 📣 Some audience members helped me answer some questions, too!

We talked a lot about office politics, tribal knowledge, and toxic workplaces. The audience generally agreed that most businesses tried to rub copious amounts of Confluence on their tribal knowledge problem, but it never improved. 😜

The room was full with people standing in the back and I’m tremendously humbled by everyone who came. I received plenty of feedback afterwards and that’s the best gift I could ever get. 🎁

Other talks #

Anita Zhang had an excellent keynote talk on the second day about her unusual path into the world of technology. Her slides were pictures of her dog that lined up with various parts of her story. That was a great idea.

Kyle Davis offered talks on valkey and bottlerocket. There was plenty about the redis and valkey story that I didn’t know and the context was useful. It looks like you can simply drop valkey into most redis environments without much disruption.

Thomas Cameron talked about running OKD on Fedora CoreOS in his home lab. There were quite a few steps, but he did a great job of connecting the dots between what needed to be done and why.

Around the exhibit hall #

I helped staff the Fedora/CoreOS booth and we had plenty of questions. Most questions were around the M1 Macbook running Asahi Linux that was on the table. 😉

There were still quite a few misconceptions around the CentOS Stream changes, as well as how AlmaLinux and Rocky Linux fit into the picture. Our booth was right next to the AlmaLinux booth and I had the opportunity to meet Jonathan Wright. That was awesome!

I can’t wait for next year’s event.

Roll your own static blog analytics

major@mhtx.net (Major Hayden) — Thu, 04 Apr 2024 00:00:00 +0000

Static blogs come with tons of advantages. They’re cheap to serve. You store all your changes in git. People with spotty internet connections can clone your blog and run it locally.

However, one of the challenges that I’ve run into over the years is around analytics.

I could quickly add Google Analytics to the site and call it a day, but is that a good idea? Many browsers have ad blocking these days and the analytics wouldn’t even run. For those that don’t have an ad blocker, do I want to send more data about them to Google? 🙃

How about running my own self-hosted analytics platform? That’s pretty easy with containers, but most ad blockers know about those, too.

This post talks about how to host a static blog in a container behind a Caddy web server. We will use goaccess to analyze the log files on the server itself to avoid dragging in an analytics platform.

Why do you need analytics? #

Yes, yes, I know this comes from the guy who wrote a post about writing for yourself, but sometimes I like to know which posts are popular with other people. I also like to know if something’s misconfigured and visitors are seeing 404 errors for pages which should be working.

It can also be handy to know when someone else is writing about you, especially when those things are incorrect. 😉

So my goals here are these:

Get some basic data on what’s resonating with people and what isn’t
Find configuration errors that are leading visitors to error pages
Learn more about who is linking to the site
Do all this without impacting user privacy through heavy javascript trackers

What are the ingredients? #

There are three main pieces:

Caddy, a small web server that runs really well in containers
This blog, which is written with Hugo and stored in GitHub
Goaccess, a log analyzer with a capability to do live updates via websockets

Caddy will write logs to a location that goaccess can read. In turn, goaccess will write log analysis to an HTML file that caddy can serve. The HTML file served by caddy will open a websocket to goaccess for live analytics.

A static blog in a container? #

We can pack a static blog into a very thin container with an extremely lightweight web server. After all, caddy can handle automatic TLS certificate installation, logging, and caching. That just means we need the most basic webserver in the container itself.

I was considering a second caddy container with the blog content in it until I stumbled upon a great post by Florin Lipan about The smallest Docker image to serve static websites. He went down a rabbit hole to make the smallest possible web server container with busybox.

His first stop led to a 1.25MB container, and that’s tiny enough for me.¹ 🤏

I built a container workflow in GitHub Actions that builds a container, puts the blog in it, and stores that container as a package in the GitHub repository. It all starts with a brief Dockerfile:

FROM docker.io/library/busybox:1.36.1
RUN adduser -D static
USER static
WORKDIR /home/static
COPY ./public/ /home/static
CMD ["busybox", "httpd", "-f", "-p", "3000"]

We start with busybox, add a user, put the website content into the user’s home directory, and start busybox’s httpd server. The container starts up and serves the static content on port 3000.

Caddy logs #

Caddy writes its logs in a JSON format and goaccess already knows how to parse caddy logs. Our first step is to get caddy writing some logs. In my case, I have a directory called caddy/logs/ in my home directory where those logs are written.

I’ll mount the log storage into the caddy container and mount one extra directory to hold the HTML file that goaccess will write. Here’s my docker-compose.yaml excerpt:

 caddy:
 image: ghcr.io/major/caddy:main
 container_name: caddy
 ports:
 - "80:80/tcp"
 - "443:443/tcp"
 - "443:443/udp"
 restart: unless-stopped
 volumes:
 - ./caddy/Caddyfile:/etc/caddy/Caddyfile:Z
 - caddy_data:/data
 - caddy_config:/config
 # Caddy writes logs here 👇
 - ./caddy/logs:/logs:z
 # This is for goaccess to write its HTML file 👇
 - ./storage/goaccess_major_io:/var/www/goaccess_major_io:z

Now we need to update the Caddyfile to tell caddy where to place the logs and add a reverse_proxy configuration for our new container that serves the blog:

major.io {
 # We will set up this container in a moment 👇
 reverse_proxy major_io:3000 {
 lb_try_duration 30s
 }

 # Tell Caddy to write logs to `/logs` which
 # is `storage/logs` on the host:
 log {
 output file /logs/major.io-access.log {
 roll_size 1024mb
 roll_keep 20
 roll_keep_for 720h
 }
 }
}

Great! We now have the configuration in place for caddy to write the logs and the caddy container can mount the log and analytics storage.

Enabling analytics #

We’re heading back to the docker-compose.yml file once more, this time to set up a goaccess container:

 goaccess_major_io:
 image: docker.io/allinurl/goaccess:latest
 container_name: goaccess_major_io
 restart: always
 volumes:
 # Mount caddy's log files 👇
 - "./caddy/logs:/var/log/caddy:z"
 # Mount the directory where goaccess writes the analytics HTML 👇
 - "./storage/goaccess_major_io:/var/www/goaccess:rw"
 command: "/var/log/caddy/major.io-access.log --log-format=CADDY -o /var/www/goaccess/index.html --real-time-html --ws-url=wss://stats.major.io:443/ws --port=7890 --anonymize-ip --ignore-crawlers --real-os"

This gets us a goaccess container to parse the logs from caddy. We need to update the caddy configuration so that we can reach the goaccess websocket for live updates:

stats.major.io {
 root * /var/www/goaccess_major_io
 file_server
 reverse_proxy /ws goaccess_major_io:7890
}

At this point, we have caddy writing logs in the right place, goaccess can read them, and the analytics output is written to a place where caddy can serve it. We’ve also exposed the websocket from goaccess for live updates.

Serving the blog #

We’ve reached the most important part!

We added the caddy configuration to reach the blog container earlier, but now it’s time to deploy the container itself. As a reminder, this is the container with busybox and the blog content that comes from GitHub Actions.

The docker-compose.yml configuration here is very basic:

 major_io:
 image: ghcr.io/major/major.io:main
 container_name: major_io
 restart: always

Caddy will connect to this container on port 3000 to serve the blog. (We set port 3000 in the original Dockerfile).

At this point, everything should be set to go. Make it live with:

docker-compose up -d

This should bring up the goaccess and blog containers while also restarting caddy. The website should be visible now at major.io (and that’s how you’re reading this today).

What about new posts? #

I’m glad you asked! That was something I wondered about as well. How do we get the new blog content down to the container when a new post is written? 🤔

As I’ve written in the past, I like using watchtower to keep containers updated. Watchtower offers an HTTP API interface for webhooks to initiate container updates. We can trigger that update via a simple curl request from GitHub Actions when our container pipeline runs.

My container workflow has a brief bit at the end that does this:

 - name: Update the blog container
 if: github.event_name != 'pull_request'
 run: |
 curl -s -H "Authorization: Bearer ${WATCHTOWER_TOKEN}" \
 https://watchtower.thetanerd.com/v1/update 
 env:
 WATCHTOWER_TOKEN: ${{ secrets.WATCHTOWER_TOKEN }}

You can enable this in watchtower with a few new environment variables in your docker-compose.yml:

 watchtower:
 # New environment variables 👇
 environment:
 - WATCHTOWER_HTTP_API_UPDATE=true
 - WATCHTOWER_HTTP_API_TOKEN=SUPER-SECRET-TOKEN-PASSWORD
 - WATCHTOWER_HTTP_API_PERIODIC_POLLS=true

WATCHTOWER_HTTP_API_UPDATE enables the updating via API and WATCHTOWER_HTTP_API_TOKEN sets the token required when making the API request. If you set WATCHTOWER_HTTP_API_PERIODIC_POLLS to true, watchtower will still periodically look for updates to containers even if an API request never appeared. By default, watchtower will stop doing periodic updates if you enable the API.

This is working on my site right now and you can view my public blog stats on stats.major.io. 🎉

Florin went all the way down to 154KB and I was extremely impressed. However, I’m not too worried about an extra megabyte here. 😉 ↩︎

Connect Caddy to Porkbun

major@mhtx.net (Major Hayden) — Thu, 29 Feb 2024 00:00:00 +0000

I recently told a coworker about Caddy, a small web and proxy server with a very simple configuration. It also has a handy feature where it manages your TLS certificate for you automatically.

However, one problem I had at home with my CoreOS deployment is that I don’t have inbound network access to handle the certificate verification process. Most automated certificate vendors need to reach your web server to verify that you have control over your domain.

This post talks about how to work around this problem with domains registered at Porkbun.

DNS validation #

Certificate providers usually default to verifying domains by making a request to your server and retrieving a validation code. If your systems are all behind a firewall without inbound access from the internet, you can use DNS validation instead.

The process looks something like this:

You tell the certificate provider the domain names you want on your certificate
The certificate provider gives you some DNS records to add wherever you host your DNS records
You add the DNS records
You get your certificates once the certificate provider verifies the records.

You can do this manually with something like acme.sh today, but it’s painful:

# Make the initial certificate request
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please

# Add your DNS records manually.

# Verify the DNS records and issue the certificates.
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please --renew

# Copy the keys/certificates and configure your webserver.

We don’t want to live this way.

Let’s talk about how Caddy can help.

Adding Porkbun support to Caddy #

Caddy is a minimal webserver and Porkbun support doesn’t get included by default. However, we can quickly add it via a simple container build:

FROM caddy:2.7.6-builder AS builder

RUN xcaddy build \
 --with github.com/caddy-dns/porkbun

FROM caddy:2.7.6

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

This is a two stage container build where we compile the Porkbun support and then use that new caddy binary in the final container.

We’re not done yet!

Automated Caddy builds with updates #

I created a GitHub repository that builds the Caddy container for me and keeps it updated. There’s a workflow to publish a container to GitHub’s container repository and I can pull containers from there on my various CoreOS machines.

In addition, I use Renovate to watch for Caddy updates. New updates come through a regular pull request and I can apply them whenever I want.

Example pull request from Renovate

Connecting to Porkbun #

We start here by getting an API key to manage the domain at Porkbun.

Log into your Porkbun dashboard.
Click Details to the right of the domain you want to manage.
Look for API Access in the leftmost column and turn it on.
At the top right of the dashboard, click Account and then API Access.
Add a title for your new API key, such as Caddy, and click Create API Key.
Save the API key and secrey key that are displayed.

Open up your Caddy configuration file (the Caddyfile) and add some configuration:

{
 email me@example.com

 # Uncomment this next line if you want to get
 # some test certificates first.
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

 acme_dns porkbun {
 api_key pk1_******
 api_secret_key sk1_******
 }
}

example.com {
 handle {
 respond "Hello world!"
 }
}

Save the Caddyfile and restart your Caddy server or container. Caddy will immediately begin requesting your TLS certificates and managing your DNS records for those certificates. This normally finishes in less than 30 seconds or so during the first run.

If you don’t see the HTTPS endpoint working within a minute or two, be sure to check the Caddy logs. You might have a typo in a Porkbun API key or the domain you’re trying to modify doesn’t have the API Access switch enabled.

Remember that Porkbun requires you to enable API access for each domain. API access is disabled at Porkbun by default.

That’s it! 🎉

Renewals #

Caddy will keep watch over the certificates and begin the renewal process as the expiration approaches. It has a very careful retry mechanism that ensures your certificates are updated without tripping any rate limits at the certificate provider.

Linux on the AMD ThinkPad Z13 G2

major@mhtx.net (Major Hayden) — Sun, 14 Jan 2024 00:00:00 +0000

AMD’s new Zen 4 processors started rolling out in 2022 and I’ve been watching for the mobile CPUs to reach laptops. I like where AMD is going with these chips and how they provide lots of CPU power without eating up the battery.

I recently ordered a ThinkPad Z13 Gen 2 with an AMD Ryzen 7. As you might expect, I loaded it up with Fedora Linux and set out to ensure that everything works.

This post includes all of the configurations and changes I added along the way.

Power management #

I removed the power profiles daemon that comes with Fedora by default. and replaced it with tlp. This is a great package for ThinkPad laptops as it takes care of most of the power management configuration for you with sane defaults. It also offers an easy to read configuration file where you can make adjustments.

The defaults seem to be working well so far, but my only complaint is that the power management for amdgpu seems to be really aggressive. Graphics performance on battery power is okay, but I’m told this improves in kernel 6.7. I’m on 6.6.11 in Fedora 39 right now.

I’ll wait to see if this new kernel makes any improvements.

Touchpad #

The ELAN touchpad in the Z13 is a bit different. It’s a haptic touchpad. It doesn’t push down with a click like the other thinkpads. It provides haptic feedback, much like a mobile phone does when you tap on the screen. (I usually turn this off on my phone, but it feels good on the laptop.)

The touchpad works right out of the box without any additional configuration. I made a basic Sway configuration stanza to get it configured with my preferences:

# ThinkPad Z13 Gen 2 AMD Touchpad
input "11311:40:SNSL0028:00_2C2F:0028_Touchpad" {
 drag disabled
 tap enabled
 dwt enabled
 natural_scroll disabled
}

The configuration above enables tap to click and dragging with taps. I like the old school scrolling style and I’ve disabled the natural scroll.

You can always get a list of your input devices in Sway with swaymsg:

swaymsg -t get_inputs

Display #

The display worked right out of the box but the UI elements were scaled up far too large for me. I typically value screen real estate over all other aspects, but my usual default of scaling to 1.0 made the UI far too small.

I set my output scaling to 1.2:

# Disable HiDPI
output * scale 1.2

I also enabled the RPM Fusion repos to get the freeworld AMD Mesa drivers:

sudo dnf install \
 https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
 https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf swap mesa-va-drivers mesa-va-drivers-freeworld
sudo dnf swap mesa-vdpau-drivers mesa-vdpau-drivers-freeworld

Audio #

Sound worked right out of the box, but I found that the loudness preset from easyeffects made the speakers sound a little bit better:

sudo dnf install easyeffects

Everything else #

Everything else just worked!

I’m really pleased with the performance and the battery life so far. My only complaint is that the OLED screen can be a battery hog at times.

For more details, check out the Arch Linux wiki page for the Z13. They documented lots of the function keys if you want to create keyboard shortcuts and they link to some downloadable monitor profiles.

Dark mode in Sway

major@mhtx.net (Major Hayden) — Tue, 09 Jan 2024 00:00:00 +0000

Ah, dark mode! I savor my dark terminals, window decorations, and desktop wallpapers. It’s so much easier on my eyes on those long work days. 😎

However, I think the author Mary Oliver said it best:

Someone I loved once gave me a box full of darkness. It took me years to understand that this too, was a gift.

In most window managers, such as GNOME or KDE, switching to dark mode involves a simple trip to the settings panels and clicking different themes. Sway doesn’t offer us those types of comforts, but we can get dark mode there, too!

GTK applications #

If you happen to have GNOME on your system alongside sway, go into Settings, then Appearance and select Dark. You can also get dark mode by applying a setting in ~/.config/gtk-3.0/settings.ini:

[Settings]
gtk-application-prefer-dark-theme=1

Restart whichever application you were using and it should pick up the new configuration.

Firefox, for example, ships with an automatic appearance setting that follows the OS. That should be reflected immediately upon restart. If not, go into Firefox’s settings, and look for dark mode under the Language and Appearance section of the general settings.

QT applications #

Most of my applications are GTK-based, but I have one or two which use QT. Again, just like the GTK example, if you have KDE installed along side Sway, you can configure dark mode there easily. Just open the system settings and look for Breeze Dark in the Plasma Style section.

You don’t have KDE? Don’t worry! There are a couple of commands which should work:

# This should work for all QT/KDE apps
# if you have the Breeze Dark theme installed.
lookandfeeltool -platform offscreen \
 --apply "org.kde.breezedark.desktop"

# You can set the theme for GTK apps here as well
# if you run into problems.
dbus-send --session --dest=org.kde.GtkConfig \
 --type=method_call /GtkConfig org.kde.GtkConfig.setGtkTheme \
 "string:Breeze-dark-gtk"

Alternate dark mode based on time #

Many window managers offer a method for adjusting dark and light modes based on the time of day. For example, some people love brighter interfaces during the day and darker ones at night. There’s a great tool called darkman that makes this easier. 🤓

The darkman service runs in the background and runs various commands to change dark mode settings for all kinds of window managers. It also speaks to dbus directly to set the configurations if needed.

It also has a directory full of user contributed scripts to change dark and light modes for various environments. You might be able to pull some commands from these files to test which configurations might work best on your system.

On diversity

major@mhtx.net (Major Hayden) — Sat, 16 Dec 2023 00:00:00 +0000

️ 👋 This post represents my own views on the topic of diversity and it doesn’t represent the views of my employer or any professional group I belong to.

I’ve written a post on diversity and deleted it several times. It remains a sensitive topic for different people for different reasons. My gut feeling is that no matter how you frame a post on diversity, some group of people will be upset about it¹.

There was a great speaker who came and spoke to us at my last job and she made an excellent point that I remember today:

Your experiences are yours. Nobody can take them away from you.

Nobody can say that your experiences do not matter.

Nobody can tell you that you didn’t experience what you experienced.

Sharing these experiences with others allows us to grow and understand more about the world around us.

That speech entirely changed my way of thinking about interactions with other people at work and at home. There are two main benefits here:

It’s incredibly freeing for someone who has experienced something to be able to share it with others and not be told that their experience was wrong or misguided.
It’s also freeing for the listener to take in someone else’s experience and be able to ask clarifying questions so they get a better understanding of how something felt for someone else.

With that in mind, here’s we go with the rest of the post. I’m not deleting it this time.

I promise. 😉

My first experience #

I’ve written in the past about my unexpected leap to lead an information security architecture team in a previous role. Being a director was a new world unto itself, but then I found that my team wasn’t performing well. To make matters worse, our success was critical to the ongoing work of the security department as a whole.

We had three members of the team that all brought something unique to the team’s perspective. All three were men of different races, but each had a different approach to security based on their backgrounds and experiences. One left the team due to some interpersonal issues that eventually boiled over.

I was suddenly down to two people and our team needed to hire two as soon as possible.

Recruiting teams started putting feelers out into the market to find talented people and I was poking several friends for referrals. A colleague in another department reached out and really wanted to join the team. I knew about her experience from several previous interactions and she was highly recommended from her peers. She joined the team and hit the ground running.

As applicants began trickling in through the recruiting team, I started with screening calls for each. We really needed someone with skills in a few key areas:

Great communicator with empathy
Knowledge of secure development and operations practices
Someone who could be trusted to operate independently and work on team projects

Most of the applicants I screened were male and that wasn’t a surprise at the time. We brought five through the screening into interviews and it was down to four males and one female. Three of them turned out to be great and they all had deep knowledge of security architecture. After another round of interviews, we began to realize that the female matched the other applicants, but her communication skills were stronger, especially under pressure.

Needless to say, we sent her the offer and she accepted! We were thrilled! Our team was full!

Getting underway #

We began chipping away at the mountain of projects set aside for our team and started making progress. Our new team member was struggling to move from the rigidity of her previous employer to our new way of working, but she adjusted well over time.

I sat down in one of our weekly leadership meetings some time later. These meetings usually involved a round-the-horn of what’s working well for each team, the threats on the board for the next few months, and our plans.

We usually had an attendee from HR in the meetings for various reasons and she asked me how our new team member was doing. I said:

Oh, she’s doing a good job. Her last company was pretty rigid and things are different here, but she’s figuring it out. She knows her stuff and she’s a team player. We’re working through some small things here and there.

Then the HR representative said:

Well, I have to commend you for building out a such a diverse team. It’s much more so than the other teams. That’s really great work and I want to make sure you’re recognized for it.

I smiled and thanked her (because that’s my usual response), but then I almost felt sick.

Different view #

I left that meeting and went back to my desk to think.

Had I assembled a diverse team intentionally? No, I didn’t. I looked for people who had the qualities we desperately needed, gave them guardrails, and got out of the way. That’s what you do with smart people, right?

Then I wondered, “Is my team really diverse?”

There were two men and two women.
Two were on the younger end of a generation and two were on the older end.
One was of mixed Asian descent, one was Hispanic, and two were what most people would likely refer to as “white.”

So maybe my team is diverse.

Then I realized that most of the people on the team had the same certifications, all had at least an undergraduate degree, and all were married. All of them were in heterosexual relationships and all dressed in a way that aligned with their gender.

Does that mean they’re not diverse?

Is my team more or less diverse than other teams?

Does any of this even matter?

Did I do a good or a bad thing?

Diversity challenges #

This brings me to the two problems I struggle with most around diversity, especially when people talk about increasing or improving diversity on their team or within their company:

Quantifying diversity is highly subjective and in the eye of the beholder.
Challenges arise when you apply diversity requirements to real world situations.

I’ll break down both of these now.

In the eye of the beholder #

You can choose how you want to measure diversity on all kinds of factors. Depending on the factors, a team can look more or less diverse. Also, your experiences often define how you judge the diversity of people and teams.

One could argue that a team made up entirely of white males is likely not very diverse. The majority of people would likely agree with that statement.

However, what if those males vary in their sexual orientations, educational backgrounds, and socioeconomic status. Is that diverse?

If you have a team of people made up of various genders with various sexual orientations from all contents on the planet, but they all went to Ive League schools and they’re all wealthy – is that diverse?

Are any of these examples diverse enough? Does the answer to that question even matter?

In my experience, assembling a team of people with different backgrounds and approaches to problems is incredibly valuable. That type of diversity led to some incredible innovation in the past.

However, these diverse backgrounds and approaches don’t always line up with differences in gender identity, socioeconomic status, sexual orientation, or other factors. This is why I find it really challenging to quantify the level of diversity within a company or in individual teams.

Rubber meets the road #

There’s a common phrase in English: “when the rubber meets the road.”

In a literal sense, it’s referring to when car tires move on pavement during a race. What it really means is, when it comes time to do something for real and the stakes are high, what happens?

Here’s another example. Let’s say you lead an engineering team that is all males and your company says that diversity must be a priority in hiring decisions.

So you take your job requisition and send it through the recruiting team. They work hard to remove any gender-specific language or anything else that might turn an applicant away. You put the job on the internet, talk to your friends about people they know, and then wait for the responses.

Let’s assume you get ten male applicants.

Do you proceed with screening and interviewing them while you try harder to drum up more female applicants? If a female applicant never appears, do you pause the hiring process while you try to find one? What if your existing applicants find other roles in the meantime and suddenly your applicant pipeline is empty?

Some might say “Yes, of course you wait until you can find a female applicant!” In that case, your team is still short-staffed and likely not performing as well as it could. Would that be good for your customers? How about your shareholders?

Others might say “No, go ahead and complete the hiring process but you should search harder for women for future roles.” In this case, you’ll have a fully staffed team and hopefully be delivering more value quickly. However, you haven’t improved the diversity on your team and that could come back to be a problem if you’re asked about it later.

Go backwards a bit with the same example and assume you get a split of ten applicants: half male and half female. That’s awesome because now you have a diverse talent pool, right?

Here’s where it gets challenging.

If you interview them all and make an offer to the female applicant because she has the skills and qualifications needed, you now have a more diverse team (on one measure) and you’re fully staffed! Great!

If you interview them all and it turns out one of the men has the best skills and qualifications, what do you do? Your company made diversity a priority, but you’re also trying to assemble a strong team.

Do you take a less qualified applicant that improves the team’s diversity?

Or, do you take a more qualified applicant that leaves the team’s diversity unchanged?

This is where diversity breaks down: when you have to really sit down and compare outcomes, there’s not a right answer.

Another viewpoint #

My wife constantly points out things to me that I completely missed and we’ve talked about this topic many times. She has asked me the same thing in the past:

Why do people in your field care so much about getting women into technology? I hate technology. Maybe other women hate technology, too. If I knew I was hired someplace because they wanted a woman for the role and they weren’t looking at how well someone could do the job, I’d be pretty upset.

She’s a medical professional and she’s happy to remind me about this:

I went to PA (physician assistant) school and most people there were women. All the nurses at my office are women. All the front office staff are women. We’re not out there trying to get male nurses or male front office staff in here all the time. We just find people who do their job well and hire them.

Our conversations really make me stop and think.

My goals #

I’d also like to see more people from underrepresented communities across the globe break into the world of technology and really change things. This means empowering a wider array of people with varying gender, education, nationality, wealth, and opportunities to join a field of work which they thought might be inaccessible to them.

This is why I try to volunteer as much as possible to inspire young people of all backgrounds to set goals for themselves and look at the world as if nothing is out of reach.

It’s one of the reasons I write this blog and put everything out there for free. Democratizing access to learning (and my mediocre blog posts) is key to leveling the playing field.

These are some of those pieces of work that are never finished.

However, I really worry that quantifying diversity or forcing one’s definition of diversity onto someone else could lead us to a bad place where no result is satisfactory. It’s much more subjective than some would like to admit and that becomes a problem when you directly apply it to specific situations.

In the meantime, I’ll keep writing these posts, mentoring others, and lifting people up to do things they never imagined they could do. ️♥️

Then again, we live in a world where someone can say “Puppies are cute” and the first reply would be “Why do you hate cats so much?” 😄 ↩︎

Horror book reviews from October 2023

major@mhtx.net (Major Hayden) — Sun, 19 Nov 2023 00:00:00 +0000

Reading allows me to travel to other places and times while also reducing my stress and helping me to think more creatively. Sometimes this leads me to wild fictional stories or takes me on a learning journey into history.

(I track my reading lists on Goodreads if you want to see what I’m reading.)

In this post, I’ll list the spooky books I read this October and hopefully you’ll find at least one them interesting!

The Cabin at the End of the World #

My first book of the month was Paul Tremblay’s The Cabin at the End of the World. It’s centered around a family with a small child that goes on a relaxing vacation in a lakefront cabin.

All is well until a friendly stranger named Leonard befriends the child, Wen, and his three fellow travelers appear. The travelers hold the family hostage and tell them that they have the key to save the world, but it’s not as easy as it seems. There seems to be an unseen force that has some sort of control over the travelers. 🤔

My thoughts #

I thought I had this story figured out several times only to be proven wrong just as many times. There are so many themes in this book that tug at your emotions, including family, racism, homophobia, and socioeconomic differences. This book packs plenty of suspense, but the most frightening parts aren’t supernatural or ghostly. The scariest parts feel centered around the essence of human nature.

Although this felt like a quick read, it was intense in many places. I definitely enjoyed it and I was looking forward to seeing the movie adaptation, Knock at the Cabin. The movie was done by M. Night Shyamalan (of The Sixth Sense fame) and I read that he changed the entire ending of the movie. 😞

The Ruins #

I moved onto something completely different next with Scott Smith’s The Ruins. The story takes place in Mexico with several people on a beach vacation. One of the tourists notes that his brother went to check out an archeological dig further into the countryside but never returned. The group eventually decides to go search for the missing brother as some sort of vacation adventure.

The adventure turns ugly as they make their way to the country’s interior. 😬

My thoughts #

You’ll have a tough time finding the antagonist in this story until it’s too late, but that’s part of the fun. Every character in the group brings personality quirks and old baggage with them that impairs their judgement in different ways. This works well for some but not for others.

This book had several scary moments, but most of the horror came again from how humans interact with one another. As soon as someone (or something) else figures out how to exploit those against them, bad things happen.

This book was difficult to read because much of it was quite gruesome and brutal. It’s definitely a book for adults only and you should be prepared to work your way through it.

The Troop #

Everything is fine with a scout trip to an island in Canada in Nick Cutter’s The Troop. Well, it’s fine until a mysterious and incredibly hungry man suddenly shows up on the island. He looks a lot like he’s dead already and the the leader of the scout troop, a medical doctor, is completely mystified by the man’s condition.

It goes downhill from there in a story told mostly through diary entries, newspaper clippings, and court testimony.

My thoughts #

Of all the books I read in October, this one scared me the most because it felt like it was entirely possible. There wasn’t any part of the book that I looked at and said: “Oh, that could never happen.” That’s what makes this one so good.

It’s suspenseful enough to keep you turning the pages but it’s also plausible enough that you might find yourself wanting to wash your hands a little longer the next time you eat a meal. It feels a bit like Lord of the Flies mixed with a pandemic novel like Station Eleven or The Stand.

This one is also very gruesome in parts with some very difficult scenes to read.

This one was my favorite of the group by far.

Devolution: A Firsthand Account of the Rainier Sasquatch Massacre #

Sasquatch is back in Max Brooks’ Devolution! Told through the journals of a woman living in a remote town in Washington, this book covers a modern time where Mount Ranier erupted and caused lots of species to get on the move from their habitats around the mountain. Some of those creatures are the ones you’d expect, but some are ones you won’t expect.

The small community is an experiment in green, off the grid living, and they are quickly tested by just about everything mother nature can throw at them. This includes some rather tall, furry, human-like creatures.

My thoughts #

This one felt scary due to the remoteness of the village and the unprepared people involved. Also, whether you believe in Sasquatch or not, this felt a bit more plausible than I expected.

There were plenty of difficult to read scenes in here, but the gore was reduced compared to other books listed here. Much of the suspense came from how humans interact with one another especially when faced with an adversary that presents a unique set of challenges.

This book was difficult to get into (it starts slow), but stick with it. It’s a wild ride.

Tender is the Flesh #

Be prepared when you crack open Agustina Bazterrica’s Tender is the Flesh. Imagine a world where animals somehow contract a virus that they quickly spread to each other and to humans as well. Pets are banned and animals near cities are killed. The pandemic puts a significant dent in the human population.

However, with all of the animals either infected or gone, where do people get meat to eat? 🤔

My thoughts #

This was by far the most challenging book to read out of the group because I honestly felt like I was going to be sick in places. The author doesn’t set out for cheap scares or basic unsettling events – there’s something much deeper. It really makes you question how humanity operates and how quickly boundaries can shift when hunger becomes a problem.

I had to take a lot of breaks with this book. It has some suspenseful parts but this book has a slow-burn horror feel that gives you hope, crushes that hope, and then starts the cycle once more.

I strongly recommend this book for adults only but I feel terrible recommending it at the same time. 😄

What’s next? #

After all of that horror and unsettling fiction, I’m shaking things up for November. My current book is Larry McMurtry’s Lonesome Dove. So many people have recommended it to me as a great book about the American west and I’m enjoying it so far.

Moving to cloud is more than just a purchasing exercise

major@mhtx.net (Major Hayden) — Fri, 27 Oct 2023 00:00:00 +0000

Much of my work at Red Hat revolves around the RHEL experience in public clouds. I thrive on input from customers, partners, and coworkers about how they consume public clouds and why they made decisions to deploy there.

Throughout this process, I run into some wild misconceptions about public clouds and what makes them useful. One that I hear most often is:

Businesses are moving to the cloud to reduce cost and improve efficiency. It’s mainly just a purchasing exercise.

This couldn’t be further from the truth.

Cloud offers a chance to start over #

Sometimes businesses find themselves in an IT quagmire¹. No matter what they do to improve their situation, it just gets worse. Capital expenditures grow and grow, datacenter space gets more expensive, and companies spend more time focusing on IT rather than their core business.

Deploying in clouds offers that chance to break the capital expense cycle and gradually improve infrastructure. The key word here is gradual.

Businesses can choose how much they want to deploy and when without worrying about expensive servers in the datacenter waiting to be used. Some deployments are greenfield, or entirely net new applications. Some are basic migrations of applications from servers or virtual machines directly to the cloud.

Either way, businesses have the freedom to deploy as little or as much as they want on their own schedule.

Cloud offers a chance to software-define (nearly) anything #

Anyone who has worked in a large organization before knows the pain of change management. Sure, it ticks a box on that yearly compliance program, but it also ensures that everyone is aligned on the plan.

One of the greatest aspects of cloud is that you can define almost everything in software. This makes changes easier to apply, easier to roll back, and easier to track.

Tools like Terraform or Ansible allow developers and operations team to work from the same playbook. My team enjoys using Infracost to track how much a particular Terraform change might cost us under different scenarios.

Once teams set a policy of “we define our changes in git, and that’s it”, you can rely on a git history for change management. This avoids drift in production environments and it also ensures that changes made in development environments make it into staging and then into production. The days of “it worked on my system, what’s wrong with production?” slowly fade away.

Less than ideal architectural decisions can also be adjusted over time to fit the applications being deployed. Did you set up a network incorrectly? Did you choose an instance type without enough RAM?

That’s okay!

Just adjust the deployment in git, test it in staging, and push it to production.

Cloud offers managed services #

One thing I tell people constantly is that if you bend the cloud to fit your application, you will almost always pay more. You get cost and performance efficiencies if you bend your application to fit the cloud. Confused? I’ll explain.

When I talk to people doing their first cloud deployments, they deploy everything into VMs, much as they would in a local virtualized environment.

You need a database server? Make a couple of VMs and set up replication.
You need to run a batch job via cron? Deploy a VM and add it to the crontab.
You need a server to export an NFS share? Deploy a VM with lots of storage and export it to other instances.

Do you see a pattern here?

Most public clouds offer tons of services that lift the management burden from engineering teams and offload into a managed service. For example, that cron job might be able to move into a “serverless”² service, such as AWS Lambda. It’s critical to check the pricing here to ensure you’re not headed down a bad path, but you have one less VM to maintain, one less IPv4 address to pay for, and a greatly reduced risk of configuration drift. That reduction in stress and risk might be worth any additional costs.

Deployment decisions become much easier and lower stress when you consume services offered by the provider. There are those situations where deploying a whole VM is needed, but I’ve managed to avoid that for some of my team’s recent deployments.

Our last deployment uses GitHub Actions, S3, and CloudFront and costs us about $6.50 per month to run. There are no virtual machines. There’s nothing to patch.

This blog runs on a similar stack and costs me about $0.25 per month to run.

Cloud offers geographic distribution #

Nearly every public cloud, even the smallest ones, offer you the same or similar services in a wide variety of geographic regions. Disaster recovery feels more attainable when you can easily deploy to multiple regions with the same software-defined infrastructure.

Data sovereignty continues to grow in importance around the world as more countries demand that their data remains within their borders. As long as your cloud offers a region in that country, you can deploy there. There’s no challenging legal issues with finding datacenter space or getting hardware delivered. You just change your region and deploy.

Cloud regions also allow you to bring your applications much closer to the people who use them. Reduced latency delivers content faster to customers and provides a responsive experience.

Clouds offer purchasing efficiency #

Wait a minute! Didn’t I say that moving to cloud isn’t just a purchasing exercise? 🤔

Your move to cloud should not be solely based on cutting costs or making purchasing IT more efficient. Most teams find that moving to cloud is more expensive than they anticipated because they’re finally able to get access to the right amount of resources that they need. (Also, they usually go with some more expensive options up front until they figure out how to optimize for cost.)

First off, it’s much easier to budget and pay one vendor for multiple services than deal with multiple independent vendors. Instead of paying for datacenter space, then paying for servers, then paying for network equipment, then paying for people to set it up, and so on, you pay the cloud provider for all of it.

This also extends to other purchases on the cloud, such as products from certain vendors. For example, you can buy Red Hat products directly from some cloud providers and that gets added onto your cloud invoice. You can even deploy your own Cisco ASA in the cloud if you feel so inclined.

With all of these purchases going through one vendor, you can also negotiate discounts if you set a spending commitment. Discounts depend on your committed spend, of course, and the term that you agree to spend it. There’s a whole industry around financial operations in the cloud, called FinOps, and this is one of many things that factors into it.

Wrapping up #

Public clouds offer an incredible amount of opportunity to get your IT deployments into better shape with better change control and a solid software-defined workflow. They also offer the ability to “write one check” to consume infrastructure via utility billing.

However, public clouds are not ideal for every application or situation.

Do I think that every company in the world could benefit from getting some part of their IT deployments into a public cloud platform? Yes, I do.

Would every company benefit from putting most of their infrastructure into public clouds? Very unlikely.

Some applications still benefit from being on purpose-built hardware or in certain locations where a cloud might not exist today. Clouds can also be extremely expensive if you run large workloads around the clock. They can also be painful for applications with very strict or special requirements that don’t fit a cloud deployment model well.

The vendors that will succeed the most in the cloud space are the ones that look beyond purchasing efficiencies and IT acquisition concerns. Simply dragging the old world of physical servers or virtual machines into cloud won’t lead anywhere.

Those companies that help their customers benefit from the best of what public clouds have to offer in the most secure, reliable, and simple ways will be in the driver’s seat.

A quagmire is something that gets worse no matter how you try to improve it. The only way to win is to avoid it entirely. ↩︎
Boy, I still dislike that serverless term so much. 🤦‍♂️ ↩︎

How I learned to stop worrying and love the CoreOS

major@mhtx.net (Major Hayden) — Fri, 13 Oct 2023 00:00:00 +0000

It’s quite clear that I’ve been on a CoreOS blogging streak lately. I keep getting asked by people inside and outside my company about what makes CoreOS special and why I’ve switched over so many workloads to it.

The answer is pretty basic. It makes my life easier.

I’m a Dad. I’m on the PTC (Parent Teacher Club) at one of my children’s schools. I volunteer as an IT person for a non-profit. I write software. I have other time consuming hobbies, such as ham radio, reading, and becoming a longer distance runner¹.

My available time for my own IT projects is extremely limited and CoreOS plays a part in keeping that part of my life as efficient as possible.

That’s what this blog post is about!

Updates #

First and foremost, I love how CoreOS does updates. I encourage you to read the docs on this topic, but here’s a short explanation:

Updates are automatically retrieved and they’re loaded into a slot.
Your system reboots into the new update but your original OS tree remains in place.
Did the update boot? Awesome. You’re good to go.
Did something break? The system reverts back to the known good tree.

In this way, it’s a lot like your smartphone.

You have full control over when a node looks for an update and how often it checks for them. Check out the Zincati docs for tons of controls over updates and reboots.

Some of mine are timed so well that I set maintenance windows with my monitoring provider when I know an update might take place. The updates come through, monitoring shuts off, the node reboots, and monitoring comes back. The nodes almost always come back before the monitoring even alerts me.

It also removes the reminders I would set for myself to update packages and run reboots. I know that my CoreOS nodes will do this automatically, so I don’t need to think about it.

Also, updates are rarely ever impactful to my workloads since all of them are running inside containers. My containers come right back up as soon as the node finishes its reboot.

Toolbox #

To be fair, you can get toolbox running on lots of different Linux distributions outside of CoreOS, but that’s the first place I ever used it. Toolbox, also called toolbx, gives you a utility container on your CoreOS node for all kinds of adminsitrative and diagnostic capabilities.

You might need a certain package for diagnosting a hardware issue or you might want to install some helpful utilities for the command line. Do that in a toolbox container. Just run toolbox enter and if you’ve never created a toolbox container before, you’ll get a Fedora container that matches your CoreOS release.

But it gets better.

Toolbox automatically saves your container when you’re done with it so all of your installed packages stay there for next time. Also, these containers have seamless access to anything you have in your home directory, including sockets. You’re running inside a container, but it’s almost like you’re running on the host itself inside your home directory. You get the best of both worlds.

Don’t want Fedora? You have lots of distribution options through toolbox. Read the details on custom images to create your own!

Layering #

Okay, there are those situations where you really want a package on CoreOS and toolbox might not be sufficient. My muscle memory for vim is so strong and CoreOS only comes with vi.

You have a couple of options here:

Run sudo rpm-ostree install vim, reboot, and you have vim
Run sudo rpm-ostree install --apply-live vim and you have vim right now! (And it’s there after a reboot as well.)

When a new update comes down for the base OS from CoreOS, any packages you’ve added will be layered on the base image and available after a reboot. Layering is generally chosen as a last resort option for adding packages to the system but you shouldn’t run into issues if you’re installing small utilities or command line tools.

Declarative provisioning #

If you’ve provisioned Linux distributions on cloud instances in the past, you’ve likely provided metadata that cloud-init uses to provision your system. CoreOS has something that acts a lot earlier in the boot process and has more power to get things done: ignition.

There’s a handy butane file forma that you use for writing your configuration. You use the butane utility to get it into ignition format. The ignition format is highly compressed to ensure you can fit your configuration into most cloud providers’ metadata fields.

For a real example of what you can do with ignition, check out my quadlets post where I provisioned an entire Wordpress container stack using a single ignition file.

There’s lots of documentation for writing butane configuration files for common situations. It’s easy to add files, configure Wireguard, set up users, and launch containers immediately on the first boot.

Pets and cattle #

CoreOS works well for systems that I only need online for a short time. These might be situations where I need to test a few containers and throw it away. There’s no OS to mess with and no updates to worry about.

It also works well for systems that I keep online for a long time. I have a few physical systems at home that run CoreOS and they’ve been extremely stable. I also have cloud instances on Hetzner, VULTR, and Digital Ocean that have run CoreOS for months without issues.

Quadlets might make me finally stop using docker-compose

major@mhtx.net (Major Hayden) — Mon, 25 Sep 2023 00:00:00 +0000

I’ve written a lot about containers on this blog. Why do I love containers so much?

They start quickly
They make your workloads portable
They disconnect your application stack from the OS that runs underneath
You can send your application through CI as a single container image
You can isolate workloads on the network and limit their resource usage much like a VM

However, I’m still addicted to docker-compose. Can podman’s quadlets change that?

Yes, I think they can.

What’s a quadlet? #

Podman introduced support for quadlets in version 4.4 and it’s a simpler way of letting systemd manage your containers. There was an option in the past to have podman generate systemd unit files, but those were unwieldy and full of podman command line options inside a unit file. These unit files weren’t easy to edit or even parse with eyeballs.

Quadlets make this easier by giving you a simple ini-style file that you can easily read and edit. This blog post will include some quadlets later, but here’s an example one for Wordpress:

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Lots of the lines under [Container] should look familiar to most readers who have worked with containers before. However, there’s something new here.

Check out the AutoUpdate=registry line. This tells podman to keep your container updated on a regular basis with the upstream container registry. I’ve used watchtower in the past for this, but it requires a privileged container and it’s yet another external dependency.

Also, at the very end, you’ll see a WantedBy line. This is a great place to set up container dependencies. In this example, the container that runs caddy (a web server) can’t start until Wordpress is up and running.

So why not stick with docker-compose? #

There’s no denying that docker-compose is an awesome tool. You specify the desired outcome, tell it to bring up containers, and it gets containers into the state you specified. It handles volumes, networks, and complicated configuration without a lot of legwork. The YAML files are pretty easy to read, too.

However, as with watchtower, that’s another external dependency.

My container deployments are often done at instance boot time and I don’t make too many changes afterwards. I found myself using docker-compose for the initial deployment and then I didn’t really use it again.

Why not remove it entirely and use what’s built into CoreOS already?

Quaint quadlets quickly! #

Before we start, we’re going to need a few things:

An easy to read butane configuration which gets transformed into a tiny ignition configuration for CoreOS
Some quadlets
Extra system configuration
A cloud provider with CoreOS images (using VULTR for this)

I’ve packed all of these items into my quadlets-wordpress repository to make it easy. Start by looking at the config.butane file.

Let’s break it down here. First up, we add an ssh key for the default core user.

variant: fcos
version: 1.5.0
passwd:
 users:
 - name: core
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0

Next up, we enable the podman-auto-update.timer so we get container updates automatically:

storage:
 links:
 - path: /home/core/.config/systemd/user/timers.target.wants/podman-auto-update.timer
 target: /usr/lib/systemd/user/podman-auto-update.timer
 user:
 name: core
 group:
 name: core

Next is the long files section:

 files:
 # Ensure the `core` user can keep processes running after they're logged out.
 - path: /var/lib/systemd/linger/core
 mode: 0644
 
 # Allow caddy to listen on 80 and 443.
 # Allow it to ask for bigger network buffers, too.
 - path: /etc/sysctl.d/90-caddy.conf
 contents:
 inline: |
 net.ipv4.ip_unprivileged_port_start = 80
 net.core.rmem_max=2500000
 net.core.wmem_max=2500000 

 # Set up an an environment file that containers can read to configure themselves.
 - path: /home/core/.config/containers/containers-environment
 contents:
 inline: |
 MYSQL_DATABASE=wordpress
 MYSQL_USER=wordpress
 MYSQL_ROOT_PASSWORD=mariadb-needs-a-secure-password
 MYSQL_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_HOST=mariadb
 WORDPRESS_DB_USER=wordpress
 WORDPRESS_DB_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_NAME=wordpress 
 mode: 0644

 # Deploy the caddy configuration file from the repository.
 - path: /home/core/.config/caddy/Caddyfile
 contents:
 local: caddy/Caddyfile
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add some named volumes for caddy and wordpress.
 - path: /home/core/.config/containers/systemd/caddy-config.volume
 contents:
 inline: |
 [Volume] 
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/caddy-data.volume
 contents:
 inline: |
 [Volume] 
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/wordpress.volume
 contents:
 inline: |
 [Volume] 
 user:
 name: core
 group:
 name: core

 # Create a network for all the containers to use and enable the
 # DNS plugin. This allows containers to find each other using
 # the container names.
 - path: /home/core/.config/containers/systemd/wordpress.network
 contents:
 inline: |
 [Network]
 DisableDNS=false
 Internal=false 
 user:
 name: core
 group:
 name: core

 # Add the wordpress container.
 - path: /home/core/.config/containers/systemd/wordpress.container
 contents:
 local: quadlets/wordpress.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the MariaDB container.
 - path: /home/core/.config/containers/systemd/mariadb.container
 contents:
 local: quadlets/mariadb.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the caddy container.
 - path: /home/core/.config/containers/systemd/caddy.container
 contents:
 local: quadlets/caddy.container
 mode: 0644
 user:
 name: core
 group:
 name: core

The Caddyfile is also in the repository and will be deployed by the butane configuration shown above.

We can go through each quadlet in detail. First up is MariaDB. We tell systemd that the wordpress container will want to have this one started first.

[Unit]
Description=MariaDB Quadlet

[Container]
Image=docker.io/library/mariadb:11
ContainerName=mariadb
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=mariadb.volume:/var/lib/mysql
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=wordpress.service multi-user.target default.target

The wordpress quadlet is much the same as the MariaDB one, but we tell systemd that caddy will want wordpress started first.

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Finally, the caddy quadlet contains four volumes and some published ports. These ports will be published to the container host. Also, you’ll note that the wordpress volume is mounted here, too. This is because caddy can serve static files much faster than wordpress can.

[Unit]
Description=Caddy Quadlet

[Container]
Image=docker.io/library/caddy:latest
ContainerName=caddy
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=caddy-data.volume:/data
Volume=caddy-config.volume:/config
Volume=/home/core/.config/caddy/Caddyfile:/etc/caddy/Caddyfile:Z
Volume=wordpress.volume:/var/www/html
PublishPort=80:80
PublishPort=443:443
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

Launch the quadlets #

There’s a launch script that ships this configuration to VULTR and launches a CoreOS instance:

#!/bin/bash
# This command starts up a CoreOS instance on Vultr using the vultr-cli
vultr-cli instance create \
 --os 391 \
 --plan vhp-1c-1gb-amd \
 --region dfw \
 --notify true \
 --ipv6 true \
 -u "$(butane --files-dir . config.butane)" \
 -l "coreos-$(date "+%s")"

To launch an instance, get your VULTR API key first. Then install vultr-cli and butane:

$ sudo dnf -y install butane vultr-cli

After launch, check to see what your containers are doing:

[core@vultr ~]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afa2d6501593 docker.io/library/caddy:latest caddy run --confi... 54 seconds ago Up 53 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp caddy
460426f39e6c docker.io/library/mariadb:11 mariadbd 35 seconds ago Up 35 seconds mariadb
92ece6538d5a docker.io/library/wordpress:fpm php-fpm 28 seconds ago Up 29 seconds wordpress

We should be able to talk to wordpress through caddy on port 80:

[core@vultr ~]$ curl -si http://localhost/wp-admin/install.php | head -n 25
HTTP/1.1 200 OK
Cache-Control: no-cache, must-revalidate, max-age=0
Content-Type: text/html; charset=utf-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Server: Caddy
X-Powered-By: PHP/8.0.30
Date: Mon, 25 Sep 2023 21:43:40 GMT
Transfer-Encoding: chunked

<!DOCTYPE html>
<html lang="en-US" xml:lang="en-US">
<head>
	<meta name="viewport" content="width=device-width" />
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
	<meta name="robots" content="noindex,nofollow" />
	<title>WordPress &rsaquo; Installation</title>
	<link rel='stylesheet' id='dashicons-css' href='http://localhost/wp-includes/css/dashicons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='http://localhost/wp-includes/css/buttons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='http://localhost/wp-admin/css/forms.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='http://localhost/wp-admin/css/l10n.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='http://localhost/wp-admin/css/install.min.css?ver=6.3.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui language-chooser">
<p id="logo">WordPress</p>

Awesome! 🎉

Managing containers #

Containers will automatically update on a schedule and you can check the timer:

[core@vultr ~]$ systemctl status --user podman-auto-update.timer
● podman-auto-update.timer - Podman auto-update timer
 Loaded: loaded (/usr/lib/systemd/user/podman-auto-update.timer; enabled; preset: disabled)
 Active: active (waiting) since Mon 2023-09-25 21:41:31 UTC; 3min 14s ago
 Trigger: Tue 2023-09-26 00:04:46 UTC; 2h 20min left
 Triggers: ● podman-auto-update.service

Sep 25 21:41:31 vultr.guest systemd[1786]: Started podman-auto-update.timer - Podman auto-update timer.

Quadlets are just regular systemd units:

[core@vultr ~]$ systemctl list-units --user | grep -i Quadlet
 caddy.service loaded active running Caddy Quadlet
 mariadb.service loaded active running MariaDB Quadlet
 wordpress.service loaded active running Wordpress Quadlet

As an example, you can make changes to caddy’s config file and restart it easily:

[core@vultr ~]$ systemctl restart --user caddy
[core@vultr ~]$ systemctl status --user caddy
● caddy.service - Caddy Quadlet
 Loaded: loaded (/var/home/core/.config/containers/systemd/caddy.container; generated)
 Drop-In: /usr/lib/systemd/user/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Mon 2023-09-25 21:46:28 UTC; 5s ago
 Main PID: 2652 (conmon)
 Tasks: 18 (limit: 1023)
 Memory: 15.1M
 CPU: 207ms

If you need to change a quadlet’s configuration, just open up the configuration file in your favorite editor under ~/.config/containers/systemd, reload systemd, and restart the container:

$ vi ~/.config/containers/systemd/caddy.container

--- make your edits and save the quadlet configuration ---

$ systemctl daemon-reload --user
$ systemctl restart --user caddy

Enjoy!

Mounting the AWS Elastic File Store on Fedora

major@mhtx.net (Major Hayden) — Wed, 13 Sep 2023 00:00:00 +0000

I package a few things here and there in Fedora and one of my latest packages is efs-utils. AWS offers a mount helper for their Elastic File System (EFS) product on GitHub.

In this post, I’ll explain how to:

Launch a Fedora instance on AWS EC2
Install efs-utils and launch the watchdog service
Create an EFS volume in the AWS console
Mount the EFS volume inside the Fedora instance

Always check the pricing for any cloud service before you use it! EFS pricing is based on how much you store and how often you access it. Backups are also enabled by default and they add to the monthly charges.

Let’s go! 🚀

Wait, what is EFS? #

When you launch a cloud instance (virtual machine) on most clouds, you have different storage options available to you:

Block storage: You can add partitions to this storage, create filesystems, or even use LVM. It looks like someone plugged in a disk to your instance. You get full control over every single storage block on the volume. An example of this is Elastic Block Storage (EBS) on AWS.
Object storage: Although you can’t mount object storage (typically) within your instance, you can read/write objects to this storage via an API. You can upload nearly any type of file you can imagine as an object and then download it later. Objects can also have little bits of metadata attached to them and some of the metadata include prefixes which give a folder-like experience. AWS S3 is a good example of this.
Shared filesystems: This storage shows up in the instance exactly as it sounds: you get a shared filesystem. If you’re familiar with NFS or Samba (SMB), then you’ve used shared filesystems already. They give you much better performance than object storage but offer less freedom than block storage. They’re also great for sharing the same data between multiple instances.

Using EFS is almost like having someone else host a network accessible storage (NAS) device within your cloud deployment.

Launching Fedora #

Every image in AWS has an AMI ID attached to it and you need to know the ID for the image you want in your region. You can find these quickly for Fedora by visiting the Fedora Cloud download page. Look for AWS in the list, click the button on that row, and you’ll see a list of Fedora AMI IDs. Click the rocket (🚀) for your preferred region and you’re linked directly to launch that instance in AWS!

I’m clicking the launch link for us-east-2 (Ohio)¹. To finish quickly, I’m choosing all of the default options and using a spot instance (look inside Advanced details at the bottom of the page).

Wait for the instance to finish intializing and access it via ssh:

$ ssh fedora@EXTERNAL_IP
[fedora@ip-172-31-2-38 ~]$ cat /etc/fedora-release 
Fedora release 38 (Thirty Eight)

Success! 🎉

Prepare your security group #

Before leaving the EC2 console, you need to make a note of the security group that you used for this instance. That’s because EFS uses security groups to guard access to volumes. Follow these steps to find it:

Click Instances on the left side of the EC2 console.
Click on the row showing the instance we just created.
In the bottom half of the screen, click the Security tab.
Look for Security groups in the security details and copy the security group ID for later.

It should be in the format sg-[a-f0-9]*.

If you click the security group name (after saving it), you’ll see the inbound rules associated with that security group. By default, items in the same security group can’t talk to each other. We need to allow that so our EFS mount will work later.

Click Edit inbound rules and do the following:

Click Add rule.
Choose All traffic in the Type column. (You can narrow this down further later.)
In the source box, look for the security group you just created along with your EC2 instance. If you took the default during the EC2 launch process, it might be named launch-wizard-[0-9]+.
Click Save rules.

Installing efs-utils #

Let’s start by getting the efs-utils package onto our new Fedora system:

$ sudo dnf -qy install efs-utils
Installed:
 efs-utils-1.35.0-2.fc38.noarch

The package includes some configuration, a watchdog, and a mount helper:

$ rpm -ql efs-utils
/etc/amazon
/etc/amazon/efs
/etc/amazon/efs/efs-utils.conf
/etc/amazon/efs/efs-utils.crt
/usr/bin/amazon-efs-mount-watchdog
/usr/lib/systemd/system/amazon-efs-mount-watchdog.service
/usr/sbin/mount.efs
/usr/share/doc/efs-utils
/usr/share/doc/efs-utils/CONTRIBUTING.md
/usr/share/doc/efs-utils/README.md
/usr/share/licenses/efs-utils
/usr/share/licenses/efs-utils/LICENSE
/usr/share/man/man8/mount.efs.8.gz
/var/log/amazon/efs

Let’s get the watchdog running so we have that ready later. The watchdog helps to build and tear down the encrypted connection when you mount and unmount an EFS volume:

$ sudo systemctl enable --now amazon-efs-mount-watchdog.service
Created symlink /etc/systemd/system/multi-user.target.wants/amazon-efs-mount-watchdog.service → /usr/lib/systemd/system/amazon-efs-mount-watchdog.service.
$ systemctl status amazon-efs-mount-watchdog.service
● amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog
 Loaded: loaded (/usr/lib/systemd/system/amazon-efs-mount-watchdog.service; enabled; preset: disabled)
 Drop-In: /usr/lib/systemd/system/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Wed 2023-09-13 18:43:46 UTC; 5s ago
 Main PID: 1258 (amazon-efs-moun)
 Tasks: 1 (limit: 4385)
 Memory: 13.3M
 CPU: 76ms
 CGroup: /system.slice/amazon-efs-mount-watchdog.service
 └─1258 /usr/bin/python3 /usr/bin/amazon-efs-mount-watchdog

Sep 13 18:43:46 ip-172-31-2-38.us-east-2.compute.internal systemd[1]: Started amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog.

Setting up an EFS volume #

Start by going over to the EFS console and do the following:

Click File systems in the left navigation bar
Click the orange Create file system button at the top right

A modal appears with a box for the volume name and a VPC selection. Select an easy to remember name (I’m using testing-efs-for-blog-post) and select a VPC. If you’re not sure what a VPC is or which one to use, use the default VPC since that’s likely where your instance landed as well.
Click Create.

There’s a delay while the filesystem initializes and you should see the filesystem show Available with a green check mark after about 30 seconds. Click on the filesystem you just created from the list and you’ll see the details page for the filesystem.

Security setup #

EFS volumes come online with the default security group attached and that’s not helpful. From the EFS filesystem details page, click the Network tab and then click Manage.

For each availability zone, go to the Security groups column and add the security group that your instance came up with in the first step. In my case, I accepted the defaults from EC2 and ended up with a launch-wizard-1 security group. Remove the default security group from each. Click Save.

Mounting time #

You should still be on the filesystem details page from the previous step. Click Attach at the top right and a modal will appear with mount instructions. The first option should use the EFS mount helper!

For me, it looks like sudo mount -t efs -o tls fs-0baabc62763375bb1:/ efs

Go back to your Fedora instance, create a mount point, and create the volume:

$ sudo mkdir /mnt/efs
$ sudo mount -t efs -o tls fs-0baabc62763375bb1:/ /mnt/efs
$ df -hT | grep efs
127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /mnt/efs

We did it! 🎉

We see 127.0.0.1 here because efs-utils uses stunnel to handle the encryption between your instance and the EFS storage system.

The disk was mounted by root, so we can add a -o user=fedora to give our Fedora user permissions to write files:

$ umount /mnt/efs
$ sudo mount -t efs -o user=fedora,tls fs-0baabc62763375bb1:/ /mnt/efs
$ touch /mnt/efs/test2.txt
$ stat /mnt/efs/test2.txt
 File: /mnt/efs/test2.txt
 Size: 0 	Blocks: 8 IO Block: 1048576 regular empty file
Device: 0,54	Inode: 17657675890899444015 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ fedora) Gid: ( 1000/ fedora)
Context: system_u:object_r:nfs_t:s0
Access: 2023-09-13 19:14:23.308000000 +0000
Modify: 2023-09-13 19:14:23.308000000 +0000
Change: 2023-09-13 19:14:23.308000000 +0000
 Birth: -

Also, efs-utils uses encrypted communication by default, which is great. There may be some situations where you don’t need encrypted communications or you don’t want the overhead. In that case, drop the -o tls option from the mount command and you’ll mount the volume unencrypted.

$ sudo umount /mnt/efs
$ sudo mount -t efs -o user=fedora fs-0baabc62763375bb1:/ /mnt/efs
$ df -hT | grep efs
fs-0baabc62763375bb1.efs.us-east-2.amazonaws.com:/ nfs4 8.0E 0 8.0E 0% /mnt/efs

Extra credit #

You can get fancy with access points that allow you to carve up your EFS storage and only let certain instances mount certain parts of the filesystem. So instance A might only be able to mount /files/hr while instance B can only mount /documents.

It would also be a good idea to take an inventory of your security groups and ensure the least amount of instances can reach your EFS volume as possible. Much of the work I did in this post was just for testing. A good plan might be to make a security group for your EFS volume and only allow inbound traffic from security groups which should access it. That would allow you to gather up all of your instances into different security groups and limit access.

Also, be aware of the EFS pricing! 💸

You are billed not only for how much storage you use, but also on requests. Different requests are priced differently depending on access frequency. Backups are also enabled by default at $0.05/GB-month!

Why Ohio? I’m mainly doing it to irritate Corey Quinn. 🤭 Any region you prefer should be fine. ↩︎

Car buying guide

major@mhtx.net (Major Hayden) — Mon, 04 Sep 2023 00:00:00 +0000

I love optimizing nearly everything in my life. Sometimes it means saving money. Other times it means squeezing every bit of performance out of a server.

But let’s try optimizing something I’ve never done on this blog before: Buying a car.

Although most of this information applies best to people in the USA, there are several things I’ve learned over the years that might benefit people in other places. Car purchases are often the second largest purchase for most Americans after buying a house. Why not optimize it as much as possible?

My family is in the market for new vehicle and I’ve immersed myself in learning more about the whole process. There are plenty of legal issues in play that many buyers don’t know about and there are tons of ways to walk into a dealership fully prepared.

Without further ado, I’ll share what I’ve learned with you!

Shopping #

Shopping and buying are two different things. You must keep them separate. If you’re not sure on the exact model of car you want, go shopping and don’t commit to buy anything.

If you have an idea of a particular car you want, go to Google and search for 5-10 competitors to that car. It’s as easy as searching “top competitors Camry” to find other cars that compete with a Toyota Camry. Go to those dealers and take a good look at the cars to see if they have the features you want. Test drive each and see if you still like them.

The really important thing here is to separate shopping from buying.

Search for prices #

Once you narrow your search down a bit, start to compare prices between dealers for different trim levels. I love using CarEdge for this since you can examine dealer inventory across the country and see how dealer prices stack up against each other.

Also check how long the vehicle has been sitting on the lot. CarEdge offers this data and it can give you an idea of how desperate a dealer is to get rid of a particular vehicle. If the total supply of a model is really high and the vehicle has been sitting on the lot longer than 90 days, you have the ability to negotiate for that car. Cars with a very low supply or cars that just arrived to the lot will likely be priced higher and dealers are less likely to budge on price.

For used cars, this is even more important since you don’t have an MSRP to work with as you do for new cars. Comparing prices for used cars is critical.

Keep in mind that some cars are in higher demand in some areas than others, too. You might be able to drive a few hours and save quite a bit. A friend of mine drove from Texas to Colorado to buy a car and saved $3,500.

Watch out for arbitrary markups! All dealers in the US are required to display a Monroney label and this shows the manufacturer’s suggested retail price. Dealers might add an addendum sticker somewhere else that show accessories they added. Sometimes these stickers show arbitrary markups from the dealer.

Since COVID, many dealers have been stacking massive fees on top of car purchases. Some of these fees exceed $10,000, $25,000 or more! This should be a massive red flag from the start of a negotiation with any dealer. If they aren’t willing to budge from their arbitrary markups, look for another dealer. 🚩

You found the car you want #

Awesome! 🥳

Whether it’s new or used, take the VIN number and dig up information about the car. You can learn a lot from just a quick Google search from the VIN!

Other than that, CarEdge has some good tools for digging up data on individual cars without too much expense. CarFax is the gold standard used and it offers some fairly inexpensive options.

Some might be saying: “Isn’t this a waste for a brand new car?” No, it’s not.

I was once buying a pickup truck that was advertised as one model year, but the truck was actually a year older. The dealer even switched the paper tags attached to the keys so I wouldn’t notice. I didn’t notice this until months later when I found it buried in my paperwork. 😡

There have also been situations where stolen cars are sitting on dealer lots. 😱

Remember, this is a big investment. Spending $20 on a CarFax report as you purchase a $30,000 car should be worth it.

Financing #

Take care of your financing before visiting a dealer. Local credit unions are often great for this but they’ve been under a squeeze lately with an increase in reposessions. Work with the credit union to get a good rate for the type of car you want.

Some people have had good luck financing through a dealership. However, getting financing set up ahead of time gives you the upper hand with financing negotiations. For example, when you sit down with in the finance office, you could say: “I have 5.9% through my local credit union. If you can beat that, I’ll finance with you.”

If you choose to go through financing with the dealer, here’s what I recommend:

Do not allow the dealer to run your credit report until the last minute. If you let them run it, they know that you’ll get a hard hit on your credit report and you’re less likely to look at other dealers. You want to leave your options open. Don’t let them run your credit until you are 100% sure you’re ready to buy from them.
Ask to see the “call sheet” and see what the bank offered the dealer for financing (the “buy rate”). It’s very likely that the dealer gets one rate from the bank and then marks it up for you. For example, the dealer might say “Oh, your rate is 8%.” Then you ask for the call sheet or the buy rate and see that they got 5.9% from the bank. You’re getting a huge markup. That’s another negotiation point.
If the dealer says they can lower your interest rate if you buy an add-on from them, such as extended warranty, STOP IMMEDIATELY. 🚨 This is called “tied selling” and is almost always illegal in the USA. This is a good moment to stop and re-evaluate the whole transaction.
You are not required to buy any of the add-ons that the finance manager offers you. Extended warranties, dent and ding protection, and tire/wheel protection are common items. There’s no requirement to purchase these. If you do decide to purchase one, be sure they can show you the actual amount that it will cost you. Don’t let them tell you how much it adds to your monthly payment. That allows them to hide the cost of certain add-ons.
You are required to pay reasonable fees for tax, title, and license. Sometimes documentation fees are rolled up into this, too. Every state is a little different on how much these cost, but you can Google “tax title and license” for your state and get a good estimation.
Demand to see the “out the door price.” If a dealer asks how much you can afford per month, don’t answer. You are interested in the full price of the car plus fees. Most states require that this comes out on a single sheet of paper with each expense clearly labeled with what you will owe for the car. If you tell a dealer “I can’t affort more than $500 per month”, then they will tinker with various parts of the deal to ensure you pay more in the long term without exceeding $500 per month. If the dealer won’t budge, keep asking them “If I was getting you a cashier’s check today, how much needs to be on the check?” They will eventually get the idea.

Dealers make the most money by far not on the lot itself, but in the finance office. Don’t get swindled there.

Get copies and read them #

You are entitled to a copy of everything that shows up in the finance office. Don’t get up from the chair until you have a copy of everything and you’ve examined each page.

Dealers commonly adjust numbers or conveniently leave out “We Owe” sheets (see the next section) in the finance office. Sometimes it’s an honest mistake. Sometimes it’s not.

The dreaded “We Owe” #

If a dealer doesn’t have something in stock that they promised you, such as an accessory or add-on, ensure it lands on the “We Owe” sheet in your paperwork.

There should be page somewhere in your sale paperwork that shows anything that the dealer owes you. In many states, this sheet must be in your paperwork even if it’s blank or zeroed out!

If a dealer promised you something and it didn’t land on the “We Owe” sheet, stop immediately and ask for that to be corrected right then.

Trade-ins #

Any time you trade in a vehicle, make it a different transaction. Allowing a dealer to add your trade-in with the current deal for purchase allows them to hide money for themselves in the deal.

First, get multiple offers from various sites that buy cars all day long. I recommend Carvana, Driveway, and Vroom. They will give you an immediate estimate online with a little bit of information. The CarEdge site I mentioned earlier also has some tools that allow you to look up your car in the Black Book, which is what dealers use to appraise cars.

Next, when you go to the dealer and they ask if you’re trading in a car, tell them you haven’t decided yet. Your best bet is to act as if you want them to talk you into it. Either way, keep the trade-in as a separate transaction.

When it comes time to talk about your trade-in, you should already have an out the door price on your car purchase. Scroll up if you’re not sure about this. 😉

Let the dealer know you have some other offers on your trade and let them know you’ll do the trade there if they can beat the offers. If they offer to beat the other deals, that’s awesome! That’s less work for you!

If they can’t, don’t worry. You can handle the trade-in separately with those other companies.

Buying online #

Finally, you can buy a car almost entirely online these days. Carvana, Driveway, and Vroom offer a fully online experience, but traditional dealers often have salespeople focused on internet deals.

This is a good way to sort out dealers who want to work with you and those that don’t. Let dealers know that:

You know what vehicle you want.
You’re comparing the offers from multiple dealers.
You’re looking to purchase within the next few weeks.
You want an out the door price on the vehicle so you know how much to get on the cashier’s check.

If they’re willing to deal with you via email or phone, that’s great! If not, there’s plenty of other dealers out there.

Further learning #

I’ve learned a lot from several YouTube channels over the years. Here are my favorites:

Car Questions Answered: Brandon runs a small used car dealership in North Carolina and gives lots of insights on what is happening the used and new car markets. He has lots of good advice on when to buy a car and which cars are the ones to avoid at certain times. If you ever wanted to go behind the scenes to see how a small used car dealership works, his channel is great.
Deshone The Auto Advisor: Deshone has lots of helpful short videos. He does offer a membership program that comes with a fee, but his short videos cover a ton of car buying and leasing recommendations. If you’re interested in leasing a car, be sure to watch some of his leasing videos.
CarEdge: CarEdge offers tons of services to help with car buying including 1:1 coaching once you have a deal sheet from a dealer. However, they also have tons of videos that explain how to buy a car effectively. They do role playing for finance managers and salespeople that highlight certain areas where buyers usually lose. Their role play videos are highly recommended.
Lucky Lopez: Lucky has tons of insight from a dealer perspective and he follows lots of trends around pricing, supply, and reposessions. Most of his content might be too much for car buyers, but it’s good information to know.
Chevy Dude: The Chevy Dude used to work for multiple dealers but now it running his own. He shares lots of sales tricks and secrets that help you prepare for making your next deal on a car – new or used.

Did I miss something? #

Let me know if I missed something and I’ll come back and edit this post! Just contact me via any of the methods below in the author block. ️⬇️

Fixing a ghost database migration failure

major@mhtx.net (Major Hayden) — Thu, 31 Aug 2023 00:00:00 +0000

I love learning about the behind the scenes aspects of just about everything. I do ham radio, I self-host lots of my personal infrastructure, and I’ve been learning more about the math behind the stock market for the last year or two.

That led me to start a blog on Ghost to share my findings with others. I started Theta Nerd¹ earlier this summer.

My deployment looked great when I started! Everything was automatically updated with watchtower and running with docker-compose on Fedora CoreOS. (Click these links to read the posts on both topics!)

However, I woke up one morning to my monitoring going off and my site was down. 😱

Why is the site down? #

Anyone who has worked in IT knows this sinking feeling. Something is down, you don’t know why, and you suspect the worst possible scenarios.

The instance hosting the blog was online and responsive, so I started digging into the logs with docker-compose logs. I suddenly found a wall of text in the logs for the Ghost container:

[2023-08-03 11:10:16] INFO Adding members.email_disabled column
[2023-08-03 11:10:16] INFO Setting email_disabled to true for all members that have their email on the suppression list
[2023-08-03 11:10:16] INFO Setting nullable: stripe_products.product_id
[2023-08-03 11:10:16] INFO Adding table: donation_payment_events
[2023-08-03 11:10:16] INFO Rolling back: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible..
[2023-08-03 11:10:16] INFO Dropping table: donation_payment_events
[2023-08-03 11:10:16] INFO Dropping nullable: stripe_products.product_id with foreign keys disabled
[2023-08-03 11:10:16] INFO Setting email_disabled to false for all members
[2023-08-03 11:10:16] INFO Removing members.email_disabled column
[2023-08-03 11:10:16] INFO Rollback was successful.
[2023-08-03 11:10:16] ERROR alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.

alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
{"config":{"transaction":false},"name":"2023-07-27-11-47-49-create-donation-events.js"}
"Error occurred while executing the following migration: 2023-07-27-11-47-49-create-donation-events.js"
Error ID:
 300
Error Code: 
 ER_FK_INCOMPATIBLE_COLUMNS
----------------------------------------
Error: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
 at /var/lib/ghost/versions/5.57.2/node_modules/knex-migrator/lib/index.js:1032:19
 at Packet.asError (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packets/packet.js:728:17)
 at Query.execute (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/commands/command.js:29:26)
 at Connection.handlePacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:478:34)
 at PacketParser.onPacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:97:12)
 at PacketParser.executeStart (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packet_parser.js:75:16)
 at Socket.<anonymous> (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:104:25)
 at Socket.emit (node:events:513:28)
 at addChunk (node:internal/streams/readable:315:12)
 at readableAddChunk (node:internal/streams/readable:289:9)
 at Socket.Readable.push (node:internal/streams/readable:228:10)
 at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

Ah, so a failed database migration in the upgrade to 5.57.2 is the culprit! 👏

I brought the site back online quickly by changing the container version for Ghost back to the previous version (5.55.2).

Why did the database migration fail? #

The error message from above boils down to this:

Error: alter table `donation_payment_events` add constraint 
`donation_payment_events_member_id_foreign` foreign key (`member_id`)
references `members` (`id`) on delete SET NULL - Referencing column
'member_id' and referenced column 'id' in foreign key constraint 
'donation_payment_events_member_id_foreign' are incompatible.

Adjusting the donation_payment_events.member_id column to be a foreign key of members.id is failing because they are incompatible types. However, as I examined both tables, both were regular varchar(24) columns without anything special attached to them:

mysql> describe members;
+------------------------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------------+---------------+------+-----+---------+-------+
| id | varchar(24) | NO | PRI | NULL | |
| uuid | varchar(36) | YES | UNI | NULL | |
| email | varchar(191) | NO | UNI | NULL | |
| status | varchar(50) | NO | | free | |
| name | varchar(191) | YES | | NULL | |
| expertise | varchar(191) | YES | | NULL | |
| note | varchar(2000) | YES | | NULL | |
| geolocation | varchar(2000) | YES | | NULL | |
| enable_comment_notifications | tinyint(1) | NO | | 1 | |
| email_count | int unsigned | NO | | 0 | |
| email_opened_count | int unsigned | NO | | 0 | |
| email_open_rate | int unsigned | YES | MUL | NULL | |
| last_seen_at | datetime | YES | | NULL | |
| last_commented_at | datetime | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| created_by | varchar(24) | NO | | NULL | |
| updated_at | datetime | YES | | NULL | |
| updated_by | varchar(24) | YES | | NULL | |
+------------------------------+---------------+------+-----+---------+-------+
18 rows in set (0.00 sec)

Going upstream #

I went to Ghost’s GitHub repository and opened an issue with as much data as I can find.

One of the first replies mentioned something about database collations. Long story short, collations describe how databases handle sorting and comparing data for different languages. Comparing some languages to other languages can be particularly challenging and this can lead to problems.

I made a switch from MariaDB to MySQL recently for the blog. Could that be related?

More searching #

I figured that I wasn’t the first one to stumble into this problem, and sure enough – I wasn’t! There’s a great blog post about a broken migration from MySQL 5 to 8 with Ghost.

In short, it required several steps to fix it:

Stop the Ghost container
Back up the database first (always a good idea)
Do a quick find/replace on the dumped database to change the collations
Drop the ghost database from the database 😱
Import the database back into MySQL
Start Ghost again

Dropping databases always makes me pause, but that’s what backups are for! 😉

How I fixed it #

In my case, my MySQL container is called ghostmysql and my Ghost database is ghostdb. Then I made a backup of the database using mysqldump:

sudo docker-compose exec ghostmysql mysqldump \
 -u root -psuper-secret-password ghostdb > backup-ghost-db.sql

Next, I copied the SQL file to another directory just in case I accidentally deleted this backup with an errant command.

cp backup-ghost-db.sql ../

Then I made a copy of the SQL file in the current directory and ran the find and replace on that copy. This changes the collations from the wrong one, utf8mb4_general_ci, to the right one, utf8mb4_0900_ai_ci²:

cp backup-ghost-db.sql backup-ghost-db-new.sql
sed -i 's/utf8mb4_general_ci/utf8mb4_0900_ai_ci/g' \
 backup-ghost-db-new.sql

Now I have the collations right for importing the database back into MySQL. But first, I have to drop the existing database. This is a good time to double check your backups!

sudo docker-compose exec ghostmysql mysql -u root \
 -psuper-secret-password
mysql> DROP DATABASE ghostdb;

Now we can import the modified backup:

cat backup-ghost-db-new.sql | sudo docker-compose exec -T \
 ghostmysql mysql -u root -psuper-secret-password ghostdb

Start all the containers:

sudo docker-compose up -d

Ghost was back online with the older version and everything looked good! I updated my docker-compose.yaml back to use latest for the Ghost version and ran sudo docker-compose up -d once more.

Within seconds, the new container image was in place and the container was running! Both migrations completed in seconds and the blog was back online with the newest version. 🎉

Theta is one of many financial Greeks that measure certain aspects of options contracts in the market. It’s also a letter in the Greek alphabet. ↩︎
The default collation in MySQL 8 is utf8mb4_0900_ai_ci. ↩︎

Open source contributions: Just do it

major@mhtx.net (Major Hayden) — Wed, 16 Aug 2023 00:00:00 +0000

I had a great time on the Fedora Podcast yesterday to talk about Fedora cloud! We talked about all kinds of Fedora-related topics, but a couple of questions came up around how to contribute, especially when there’s not a lot of structure in place for a particular type of contributions. Here’s the full video if you’re interested:

That made me think about a post that deserves to be written: How do you get started with open source contributions in a new project? 🤔

My answer is pretty simple: Just do it. 🚀

Just do what? #

In the late 1980’s, one of Nike’s ad agencies came up with the phrase as a way to push through uncertainty. I was pretty young when this campaign started, but the general idea was this:

Anyone can achieve what they want
Stop worrying about whether you can actually do something
Try something new
Just do it

Simple, right?

This works for open source contributions, too. I often have conversations with people inside and outside of work where they identify a problem or an improvement in an open source project. My customary response is “Let’s go upstream and make this better!”

However, what I hear back most often is “I don’t know how.” This is where the whole just do it part comes in.

I found a bug #

Nearly every open source project wants to know about bugs that users experience. Start by finding out the best way to communicate with the people working on a particular project.

For projects on GitHub or GitLab, you can open up an issue and describe your problem. Some repositories have a template generated for bugs that ask you several important questions, so be sure to follow those templates. If there isn’t a template, I usually follow this format:

What happened that was unexpected?
What were you doing right before that unexpected event happened?
What did you expect to happen instead?
What else is nearby in the environment that might have an impact?
- For example, the versions of Python might be important for Python-based projects.
What log files or other diagnostic materials exist?

What’s the goal? We want to give maintainers enough information for a quick diagnosis in the best case. If it’s not obvious, then they need enough information to try to reproduce it on their own machine for debugging.

Maintainers might come back with additional questions about your environment or the events just before the bug occurred. Be sure to respond in a timely way while the information is top of mind for them.

Always remember that these maintainers are real people who are likely not being paid for their work. Assume the best of intentions (unless proven otherwise) and stay focused on the solution. There might always be the chance that the maintainers are not interested in your use case and might not be interested in solving it.

That leads me to the next step.

I found a bug and I want to fix it #

Start by opening an issue or a bug report first (see the previous section).

This ensures that maintainers get a full picture of the problem you’re trying to solve. Also, I’ve had maintainers immediately reply and tell me that it’s a known issue already being solved in another issue. That could save you some work.

If you have a patch that fixes the issue, go through the following steps before submitting the fix upstream:

Ensure your fix references the issue or bug report that you opened
Use a very clear first line in your commit message, such as parser: Fix emoji handling in YAML rather than Fix YAML bug
Include a very brief explanation of the bug you’re fixing in the commit message
Extra credit: Add or update existing tests so they catch the bug you just found
Extra credit: Add or update the project documentation for your change if necessary

These extra credit items often make it easier to review your patch. Maintainers love extra test coverage, too.

Submit your change in a pull request or merge request and watch for updates. Be patient with replies from the maintainers, but be timely in your replies. Remember that your use case might be an edge case for the upstream project and you might need to explain your fix (or the original bug) in more detail.

I want to improve something #

Improving an open source project could involve several things, such as:

Enhancing by adding a new feature
Optimizing an existing feature
Creating documentation
Building integrations

I strongly recommend opening an issue first with the project maintainers to explain your enhancement. These Requests for Enhancements, or RFEs, should include several things:

Your use case that made you think of the enhancement in the first place
What you plan to add, substract, or change
How the changes might affect different users, especially as they upgrade from older versions
How the changes might affect testing or release processes
Any changes in dependencies required

Before going down the road of enhancements, always bring up these ideas with the maintainers first. You want to ensure that your ideal changes are aligned with the future goals of the project. In addition, maintainers will want to better understand your use case.

Remember that an enhancement almost always requires additional work from maintainers. Every new use case means more work to ensure the project still functions. That’s why it’s critical to share your use case and have a good plan for testing and documentation.

Getting involved #

Whenever I find an open source project that I’d like to get involved with, I start looking around for several things:

What do they use for informal asynchronous chat? IRC? Matrix? Slack? Something else? I join the chat, introduce myself, and get an idea for how they interact. Some groups are very chatty and informal while others are much more formal and regimented.
Where do they have detailed discussions? Many projects have detailed discussions in their issues/bugs or in places like GitHub’s discussions. Others use old school mailing lists. Some groups have regular meetings where anyone can add agenda items for discussions. If I need to talk about something a bit more long form and I expect some back and forth on it, I look for this avenue.
What requirements exist for contributors? Some projects require that contributors sign a CLA or some other sort of agreement. Make sure that any CLAs you sign are approved by your employer (if applicable). You might need an account on a system that you don’t have, so check for that as well.

From there, I take the just do it mentality and go for it. The worst thing you’ll be told is “No”. If that happens, take a step back, see if there’s another way to approach it, and try again.

Remember one thing most of all: avoid taking anything personally. All of us have our bad days and some people have personalities that might be totally incompatible with yours (and most people in general). 🤭

Add CloudFront CDN to a Ghost blog

major@mhtx.net (Major Hayden) — Mon, 03 Jul 2023 00:00:00 +0000

After I launched my new stock market blog on a self-hosted Ghost, I wrote up the deployment process in containers last week. Then I had a shower thought: How do I put a CDN in front of that?

This blog is back on an S3 + CloudFront deployment at AWS and I figured CloudFront could work well for a self-hosted Ghost blog, too.

There are tons of blog posts out there that have outdated processes or only show you how to do one piece of the CDN deployment for Ghost. I read most of them and cobbled together a working deployment. Read on to learn how to do this yourself!

Why add a CDN? #

Content Delivery Networks (CDN) enhance websites by doing a combination of different things:

High throughput content delivery. CDNs have extremely well connected systems with plenty of bandwidth available. When your web traffic goes overboard or a popular person links to your site, CDNs allow you to continue serving content at very high rates.
Cached content. CDNs will pull content from your origin server (the one running your application) and cache that content for you. This means fewer requests to your origin server and less bandwidth consumed there.
Content closer to consumers. You might host your site in the eastern USA, but a CDN can cache your content around the world for faster access. Your website might normally be slow for someone in Tokyo, but a local CDN endpoint in Japan could serve that content immediately there.
Improved security. Many CDNs offer a web application firewall (WAF) that allows you to limit access to certain functions on your site. This could prevent or slow down certain types of attacks that could take your site offline.

CDNs have trade-offs, though. They’re complicated.

They often require lots of DNS changes. TLS certificates remain a challenge. Caching solves lots of problems but can create headaches in a flash. A misconfiguration at the CDN level can take down your site or prevent it from operating properly for longer periods of time.

Careful planning helps a lot! Measure twice, cut once.

AWS terminology #

The names of various AWS services often confuse me, but here’s what we need for this project:

AWS Certificate Manager: handles TLS certificate issuance and renewal for the CDN distribution
AWS CloudFront: the actual CDN itself

CloudFront has a concept of distributions, which is a single configuration of the CDN for a particular site. We will get to that in the CloudFront section. 😉

Certificates #

First off, we need a certificate for TLS connections. Run over to the AWS Certificate Manager (ACM) console for your preferred region and follow these steps:

Click the orange Request button at the top right.
Request a public certificate on the next page and click Next.
Type in the domain for your certificate that your users will type to access your site. For example, example.com or blog.example.com.
Click Request

You should be back to your certificate list. Refresh the page by clicking on the circle with the arrow at the top right. Click on the certificate for the domain name you just added.

In the second detail block labeled Details, look for the CNAME name and value at the far right. You need to set both of these wherever you host your DNS records. If you use AWS Route 53 for DNS, there’s a button you can click there to do it immediately. If you use another DNS provider, create a CNAME record with the exact text shown there.

Once you create those DNS records, go back to the page with your certificate and wait for it to change from Pending validation to Issued. This normally takes 2-3 minutes for most DNS providers I use.

Wait for this to turn green and say Issued before proceeding to the next step!

Now that you have a certificate, it’s time to configure our CDN distribution.

CloudFront #

Now comes the fun, but complicated part. You have two DNS records to think about here:

The CDN DNS record that users will type to access your site, such as example.com.
The origin DNS record that the CDN will use to access your backend Ghost blog, such as origin.example.com.

The origin record will be hidden away behind the CDN when we’re done.

Create the distribution #

Go to the CloudFront console in your preferred region and follow these steps:

Click Create Distribution at the top right.
Put your origin (hidden) domain in Origin domain, such as origin.example.com.
Skip down to Name for the distribution such as “My Ghost Blog”. (This is for your internal use only.)
Compress objects automatically: Yes
Viewer protocol policy: Redirect HTTP to HTTPS
Allowed HTTP methods: GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
Cache policy: CachingOptimized
Origin request policy: AllViewerExceptHostHeader
WAF: Do not enable security protections (This costs extra and you can tweak this configuration later if needed.)
Alternate domain name (CNAME): Use the DNS name that your users will access, such as example.com
Custom SSL certificate: Choose the certificate we created in the previous section
Click Create distribution

This can take up to 10 minutes to deploy once you’re finished. At this point, we have an aggressive caching policy that will cause problems when members attempt to sign in or manage their membership. It will also break the Ghost administrative area.

Let’s fix that next.

Adjust caching #

Find the CloudFront distribution we just created and click the Behaviors tab. We are going to make three different sets of behavior configurations to handle the dynamic pages.

Click Create Behavior and do the following:

Enter /ghost* as the path pattern.
Choose the origin from the drop down that you specified when creating the distribution.
Compress objects automatically: Yes
Viewer protocol policy: Redirect HTTP to HTTPS
Allowed HTTP methods: GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
Cache policy: CachingDisabled
Origin request policy: AllViewer
Click Save changes

That takes care of the administrative interface. Now let’s fix the caching on the members page:

Enter /members* as the path pattern.
Choose the origin from the drop down that you specified when creating the distribution.
Compress objects automatically: Yes
Viewer protocol policy: Redirect HTTP to HTTPS
Allowed HTTP methods: GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
Cache policy: CachingDisabled
Origin request policy: AllViewer
Click Save changes

With this configuration, we have caching for all content except for the administrative and member interfaces.

Testing #

There are a few different ways to test at this point, but I prefer to go with an old tried and true method: the /etc/hosts file. 😜

CloudFront offers a domain name on *.cloudfront.net that you can use, but it’s not quite the same. Cookies for the admin/member interface don’t always work since they cross domains and sometimes you’re redirected back to the original domain name which bypasses the CDN altogether.

Go back to the list of distributions in your CloudFront console in your preferred region. Click on the distribution you created earlier. At the top left, you’ll see Distribution domain name with a domain underneath that contains random_text.cloudfront.net.

Take that domain name and get an IPv4 address:

$ dig +short A d2xznlk9a1h8zn.cloudfront.net 
18.161.156.2
18.161.156.18
18.161.156.61
18.161.156.9

Open /etc/hosts in your favorite editor (root access required) and use one of the IP addresses that correspond to your CDN endpoint. Add a line like this one (using your CDN domain and IPv4 address from the last step):

18.161.156.2 example.com

Access your site in a browser and verify that everything works. Be sure that you can access the administrative console under example.com/ghost and any member settings.

️Remove the line in /etc/hosts now that we’re finished with testing.

Production #

Our first step is to set up the origin.

Origin configuration #

Ensure your origin server has a proper DNS record so that CloudFront can access it on the backend. For example, origin.example.com must have a DNS record that points to your backend server running Ghost.

Verify that the DNS record for your origin works before proceeding. 💣

If you followed my guide for deploying Ghost, then you need to adjust your caddy configuration to answer requests to your origin URL. I updated my Caddyfile to contain both the origin and CDN hostnames:

{
 email major@mhtx.net
}
thetanerd.com, origin.thetanerd.com {
 reverse_proxy ghost:2368
 log {
 output stderr
 format console
 }
}

www.thetanerd.com {
 redir https://thetanerd.com{uri}
}

Restart caddy with sudo docker-compose restart caddy.

Verify that caddy responds to requests to the origin hostname before going any further. It must respond properly with a valid SSL/TLS certificate! 💣

Big switch #

Now that our origin server is happy and responding, it’s time to make the big switch. We’re going to remove the record for the main CDN domain, such as example.com and replace it with a CNAME or ALIAS record to the CDN name in CloudFront. This is the name that ends in cloudfront.net that we used for testing earlier.

The use of a CNAME or ALIAS record depends on your DNS host and the type of domain name you’re using for the CDN.

If you’re using apex domain name (no subdomain) such as example.com, you will likely need to use an ALIAS record
For domain names with a subdomain, such as blog.example.com, you will likely need to use a CNAME record

Read your DNS host’s documentation if you are unsure about ALIAS vs CNAME records! 💣

Go your DNS registrar and follow these steps:

Screenshot your existing DNS records or export them if possible (in case you need to revert).
Remove the existing A/AAAA/CNAME/ALIAS record(s) for your main domain name, such as example.com.
Immediately add a CNAME/ALIAS record from example.com to random_text.cloudfront.net that corresponds to your CloudFront distribution.

Once that’s done, I usually run curl in a terminal to watch for the changeover with watch curl -si https://example.com. When CloudFront is handling your traffic you’ll see headers like these:

HTTP/2 200 
content-type: text/html; charset=utf-8
cache-control: public, max-age=0
date: Mon, 03 Jul 2023 19:44:55 GMT
server: Caddy
x-powered-by: Express
etag: W/"19e7b-q5fZSjf8acC7o9lhdO5R+jOASfM"
vary: Accept-Encoding
x-cache: Miss from cloudfront
via: 1.1 b2ba542a917451d9d85e07dba0cfd9a4.cloudfront.net (CloudFront)
x-amz-cf-pop: DFW57-P2
x-amz-cf-id: Tpcjk886L0xAZzOjuUP-js_7-twE7ZGDZKlkmGHNTjW8hEs7oOWaLg==

If it seems like it’s taking a very long time to change over, use a tool like DNS Checker to see how various DNS servers see your recent DNS change.

Revert (if needed) #

If something went horribly wrong, DON’T PANIC. 😱

DNS is like IT quicksand. Once you get stuck in a problem with DNS, any level of fighting just makes you more stuck. Take a deep breath first. 🫁

Go back to your DNS provide and remove the ALIAS/CNAME record for your CDN domain name, such as example.com. Add back in the original A/AAAA/ALIAS/CNAME records that were there previously. Be patient for traffic to shift back to your origin server.

Review the changes you made and look for any errors.

Configuring Ghost #

Ghost is fairly easy to put behind a CDN, but it does have some additional caching configuration that you can change if needed. It provides hints to the CDN about what should and should not be cached and for how long. Refer to the Ghost docs for details.

I decided to cache requests to the Content API and to the frontend for 60 seconds as a test. My docker-compose.yml now looks like this:

 ghost:
 image: docker.io/library/ghost:5
 container_name: ghost
 restart: always
 depends_on:
 - ghostdb
 environment:
 url: https://thetanerd.com
 caching__contentAPI__maxAge: 60
 caching__frontend__maxAge: 60
 database__client: mysql
 database__connection__host: ghostdb
 database__connection__user: ghost
 database__connection__password: ...
 database__connection__database: ghostdb
 volumes:
 - ghost:/var/lib/ghost/content

Now if I access the main page of the site, I see cache hits in the headers:

HTTP/2 200 
content-type: text/html; charset=utf-8
cache-control: public, max-age=600
date: Mon, 03 Jul 2023 19:54:39 GMT
etag: W/"19e7b-5MKnFrme/sGk5DT2yvMkbgDsl+4"
server: Caddy
x-powered-by: Express
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 308bae6dc9384ec8e0a82ba2d96014bc.cloudfront.net (CloudFront)
x-amz-cf-pop: DFW57-P2
x-amz-cf-id: 0Dvoc_ST8-FK_TD4lEMQg6-uiDqhaUbYAqbylkiUP61eGcQsZSFEGg==
age: 7

The x-cache header shows a hit and the age header says it’s been cached for 7 seconds.

Enjoy your new CDN-accelerated Ghost blog! 🐇

Deploy a containerized Ghost blog 👻

major@mhtx.net (Major Hayden) — Tue, 27 Jun 2023 00:00:00 +0000

There’s no shortage of options for starting a self-hosted blog. Wordpress might be chosen most often, but I stumbled upon Ghost recently and their performance numbers really got my attention.

I prefer deploying most things in containers these days with Fedora CoreOS. Luckily, the Ghost stack doesn’t demand a lot of infrastructure:

Ghost itself
MySQL 8+ (I went with MariaDB 11.x)
A web server out front
TLS certificate

Although I chose MariaDB for the database here, Ghost recommends MySQL and will throw a warning in the admin panel if you’re using something else. I haven’t had any issues so far, but you’ve been warned. 💣

I picked Caddy for the webserver since it’s so small and the configuration is tremendously simple.

Launch CoreOS #

Fedora CoreOS offers lots of cloud options for launching it immediately. Many public clouds already have CoreOS images available, but I love Hetzner’s US locations and I already had a CoreOS image loaded up in my account.

🇩🇪 Want CoreOS at Hetzner? There’s a blog post for that!

Once your CoreOS instance is running, connect to the instance over ssh and ensure the docker.service starts on each boot:

sudo systemctl enable --now docker.service

This ensures that containers come up on each reboot. CoreOS has a podman socket that listens for docker-compatible connections, but that doesn’t help with reboots.

Perhaps I’m old fashioned, but I still enjoy using docker-compose for container management. I like how I can declare what I want and let docker-compose sort out the rest.

Let’s install docker-compose on the CoreOS instance now:

# Check the latest version in the GitHub repo before starting!
# https://github.com/docker/compose
curl -LO https://github.com/docker/compose/releases/download/v2.19.0/docker-compose-linux-x86_64

# Install docker-compose and make it executable.
sudo mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
sudo chown +x /usr/local/bin/docker-compose

Verify that docker-compose is ready to go:

$ docker-compose --version
Docker Compose version v2.19.0

Preparing Caddy #

Caddy uses a configuration file called a Caddyfile and we need that in place before we deploy the other containers. Within my home directory, I created a directory called caddy:

mkdir caddy

Then I added the Caddyfile inside the directory:

{
 # Your email for LetsEncrypt warnings/notices.
 email youremail@domain.com 

 # Staging LetsEncrypt server to use while testing.
 # Uncomment this before going to production!
 acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

# Basic virtual host definition to feed traffic into the
# Ghost container when it arrives.
example.com {
 reverse_proxy ghost:2368
}

# OPTIONAL: Redirect traffic to 'www' to the bare domain.
www.example.com {
 redir https://example.com{uri}
}

This configuration sets up LetsEncrypt certificates automatically from the staging server for now. Once we know our configuration is working well, we can comment out the acme_ca line above and get production TLS certificates.

At this point, you need a DNS record pointed to your server so you can get a certificate. You have some options:

If the site is entirely new, just point the root domain name to your CoreOS instance. Use that domain in the configuration above and later in the deployment.
If you’re migrating from an existing site, choose a subdomain off your main domain to use. If your website is example.com, use something like test.example.com or new.example.com to get Ghost up and running. It’s really easy to change this later.

Now we’re ready for the rest of the deployment.

Deploying containers #

Here’s the docker-compose.yml file I’m using:

---
version: '3.8'
services:

 # OPTIONAL
 # Watchtower monitors all running containers and updates
 # them when the upstream container repo is updated.
 watchtower:
 image: docker.io/containrrr/watchtower:latest
 container_name: watchtower
 restart: unless-stopped
 hostname: coreos-ghost-deployment
 environment:
 - WATCHTOWER_CLEANUP=true
 - WATCHTOWER_POLL_INTERVAL=3600
 command:
 - --cleanup
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 privileged: true

 # Caddy acts as our external-facing webserver and handles
 # getting TLS certs from LetsEncrypt.
 caddy:
 image: caddy:latest
 container_name: caddy
 depends_on:
 - ghost
 ports:
 - 80:80
 - 443:443
 restart: unless-stopped
 volumes:
 - ./caddy/Caddyfile:/etc/caddy/Caddyfile:Z
 - ghost:/var/www/html
 - caddy_data:/data
 - caddy_config:/config

 # The Ghost blog software itself
 ghost:
 image: docker.io/library/ghost:5
 container_name: ghost
 restart: always
 depends_on:
 - ghostdb
 environment:
 url: https://example.com
 database__client: mysql
 database__connection__host: ghostdb
 database__connection__user: ghost
 database__connection__password: GHOST_PASSWORD_FOR_MARIADB
 database__connection__database: ghostdb
 volumes:
 - ghost:/var/lib/ghost/content

 # Our MariaDB database
 ghostdb:
 image: docker.io/library/mariadb:11
 container_name: ghostdb
 restart: always
 environment:
 MYSQL_ROOT_PASSWORD: A_SECURE_ROOT_PASSWORD
 MYSQL_USER: ghost
 MYSQL_PASSWORD: GHOST_PASSWORD_FOR_MARIADB
 MYSQL_DATABASE: ghostdb
 volumes:
 - ghostdb:/var/lib/mysql

volumes:
 caddy_config:
 caddy_data:
 ghost:
 ghostdb:

I love watchtower but that step is completely optional. It does require some elevated privileges to talk to the podman socket, so keep that in mind if you choose to use it.

Our ghostdb container starts first, followed by ghost, and then caddy. That follows the depends_on configuration keys shown above.

There are two steps to take now:

Replace GHOST_PASSWORD_FOR_MARIADB and A_SECURE_ROOT_PASSWORD above with better passwords. 😉
Also, set the url parameter for the ghost container to your blog’s domain name.

Once all of that is done, let’s let docker-compose do the heavy lifting:

sudo docker-compose up -d

Let’s verify that our containers are running:

$ sudo docker-compose ps
NAME IMAGE COMMAND SERVICE 
caddy caddy:latest "caddy run --config …" caddy 
ghost docker.io/library/ghost:5 "docker-entrypoint.s…" ghost 
ghostdb docker.io/library/mariadb:11 "docker-entrypoint.s…" ghostdb 
watchtower docker.io/containrrr/watchtower:latest "/watchtower --clean…" watchtower

Awesome! 👏

Ghost initial setup #

With all of your containers running, browse to https://example.com/ghost/ Just add /ghost/ to the end of your domain name to reach the admin panel. Create your admin account there with a good password.

If everything looks good, run back to your Caddyfile and comment out the acme_ca line:

{
 # Your email for LetsEncrypt warnings/notices.
 email youremail@domain.com 

 # Staging LetsEncrypt server to use while testing.
 # Uncomment this before going to production!
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

Restart the caddy container to get a production LetsEncrypt certificate on the site:

sudo docker-compose restart caddy

Customizing Ghost #

Ghost looks for lots of environment variables to determine its configuration and you can set these in your docker-compose.yml file. Although some configuration items are easy, like url, some are nested and get more complicated. For these, you can use double underscores __ to handle the nesting.

As an example, we already used database__connection__host in the docker-compose.yaml, and that’s the equivalent to this nested configuration:

"database": {
 "connection": {
 "host": "..."
 }
}

If you’re deploying in containers, it’s a good idea to configure Ghost via environment variables. This ensures that your docker-compose.yml is authoritative for the Ghost deployment. You can exec into the container, adjust the config file on disk, and restart Ghost, but then you have to remember where you configured each item. 🥵

Switching to production domain #

If you used a temporary domain to get everything configured and you’re ready to use your production domain, follow these steps:

Open your Caddyfile and replace all instances of the testing domain with the production domain
Restart caddy: sudo docker-compose restart caddy
Edit the docker-compose.yml and change the url key in the ghost container to the production domain
Apply the configuration with sudo docker-compose up -d

Enjoy your new automatically-updating Ghost blog deployment! 👻

Engineering through layoffs

major@mhtx.net (Major Hayden) — Sun, 25 Jun 2023 00:00:00 +0000

All comments and thoughts in this post are my own and certainly do not reflect the positions of any of my employers, past and present. The goal of this post is to help with healing after a layoff event and organizing your thoughts around your decisions afterwards.

Whatever you want to call them – layoffs, reductions in force, or downsizing – they’re terrible.

For those who leave, uncertainty can become overwhelming. Loss of work means a loss of salary on the simplest level, but it can also mean a loss of purpose. It can mean a loss of critical medical insurance benefits.

For those who stay, layoffs shake the foundations of trust with the employer. I have a whole blog post on red flags that goes into detail on this topic, so I won’t repeat it all here.

My argument in this post is that for those who stay and recommit to the mission of the business, you have much more control over customer outcomes than you ever did before.

How to think about layoffs #

I went through several rounds of layoffs at my last employer and my current employer just did a round. Engineers around me often say things like:

“How could they let him go? He was so helpful!”
“She is critical to this project and it’s our top priority. How could she leave now?”
“He was there for 24 years and he helped everyone, even our CEO. Why would they make him go?”
“Our quarterly results looked great. Why does anyone need to be laid off?”

The first step is to avoid reading deeply into the decision and avoid making it personal.

In my experience, some of the decisions are made based on data that you can’t see at a publicly traded company. Sometimes a company sheds employees simply because everyone in their market sector is doing it and they’re looking for a temporary bump in the stock price. And then there are those situations where a company chooses to end a product line or project.

These decisions are often made at a high level within the company and done in such a way to avoid any type of employment-related lawsuits. That brings me to my next topic.

Why do they let top performers go? #

This frustrated me many times over the years. As an example, there was a talented network engineer on a team who was a rising star in the company. He could work through complex network topologies to provide a balance of performance and security based on the customer’s demands.

Even better, he could explain it all to customers. Better still, he could explain it to the customer’s technical and non-technical staff.

The customer was getting closer to making the deal and this engineer was central to the deal being made. The deal was large. Everyone was preparing with implementation calls, documents, and everything else needed for the final meeting.

The final meeting came, but the network engineer wasn’t on the call.

Salespeople, solutions architects, and other engineers were frantic. Did he go home sick? Did we send him the wrong time on the calendar invitation?

No, as it turns out, he was laid off at lunch and the call was scheduled for 2PM.

A choice was made to reduce engineering staff by some percentage, so the business essentially did this:

SELECT *
FROM employees
WHERE job_family = "engineering"
ORDER BY RAND()
LIMIT 100

And they went through the list methodically until their percentage was met.

This is why taking layoffs personally will only cause you pain. Avoid looking for a deeper meaning and explanation where one does not exist.

As Yoda once said:

Fear is the path to the dark side. Fear leads to anger. Anger leads to hate. Hate leads to suffering.

How about the people who aren’t top performers?

Why don’t companies just lay off low performers? #

(Again, try to to avoid looking for deeper meaning here, but I’ll go through this question anyway.)

I’ve heard this many times and I’ve asked it myself before:

Why don’t companies just lay off the low performers? After all, some of these people might be toxic to teams and it’s clear they’re not committed to the company mission.

It’s a good argument! If a company wants to save money by spending less on salaries and benefits, why not target the people who aren’t doing the work first? You’d reduce expenses while improving the quality of the workforce!

Long story short: It’s not that simple.

A wise manager once told me that:

If you’re the last person to find out that your performance is inadequate, that’s not your fault. It’s your manager’s fault.

Managers make mistakes. Whether they’re mistakes made on purpose or not, it often opens the company up to litigation.

For example, a manager might label an employee as a low performer due to factors outside their job performance. Perhaps they don’t look like the rest of the team, they have a different religious affiliation, or a different sexual orientation. They might not participate in after-work functions with the team where alcohol is involved. In some extreme situations, an employee might be labeled a low performer due to rejected romantic advances from the manager. (This last one seems crazy, but I’ve seen it happen once.)

The problem shows up when the company tries to do a layoff like this:

SELECT *
FROM employees
WHERE performance_level = "unacceptable"
ORDER BY RAND()
LIMIT 100

Suddenly there are people on the low performing list who don’t belong there. However, at the executive level, they have no idea about the dubious performance reviews.

This is a fast path to wrongful termination lawsuits.

First off, be sure that you’re ready to recommit to the company mission. If something happened that shook your commitment to the core, take some time to truly understand how you feel about your company. My post on red flags might help.

Let’s get back on something positive. How do we avoid taking these events personally and push through to something better?

Use your newfound power #

As an engineer, you have more control over customer outcomes after a layoff than ever before. Confused? I’ll explain.

I’ve worked in engineering, management, and leadership roles in technology since 2004. In many situations, engineers struggle to change business processes and persuade business-minded people to change their outlook on a topic. There’s another blog post on here about persuasion engineering that might be worth reading.

Layoffs shake the foundations of any company, including the processes that brought the company to that point. It’s a great time to question any of these processes. Does the process save time? Does it benefit customers? Does it need to be modified? Should we throw it away completely?

I’m not suggesting that you approach all processes and business justifications with immediate contempt, but have the courage to ask questions about them. Even long-held beliefs should be questioned.

For example, I recently had an exchange like this one:

Me: “What if we offered customers the capability to do X?”
Them: “Well, we don’t have any data to support that.”
Me: “This could be an opportunity to guide customers to doing X on our Y product.”
Them: “But we need something well defined that customers have asked for before going down that path.”
Me: “We’ve followed that data for quite some time and the uptake from customers is low. We just went through a round of layoffs – perhaps we should take a leap here and try something new?”

The number one fear I have as someone who stays when a layoff happens is this: What if we’re too afraid to speak up? What if we’re too afraid to take a leap? What if fear of being next on the layoff list prevents us from doing something amazing?

You can be an advocate for change. It’s the best environment to make a change and think differently about where the company can best serve its customers.

It could end in one of two ways:

You change the future of the company for the better and delight your customers
You’re on the termination list for the next layoff

On the first one, you’ve done something truly incredible and you will likely receive recognition for it. You’ll also feel more engaged in your work.

On the second one, if the company decides you rocked the boat too much and decides to let you go, it’s for the best. You’re likely dealing with some levels of middle management who lead with fear rather than a drive to improve. Don’t take it personally and look for the next opportunity.

Personally, I’d rather go out in a blaze of glory trying to make the company a better place. 😉

Launch a watchtower container via podman quadlets

major@mhtx.net (Major Hayden) — Wed, 31 May 2023 00:00:00 +0000

Most of my container workloads run on independent CoreOS cloud instances that I treat like pets. Keeping containers update remains a constant battle, but it’s still easier than running kubernetes.

I wrote about using watchtower in the past to keep containers updated. It’s a simple container that does a few important things:

It monitors (via docker/podman socket) the running containers on the host
It tracks the versions/tags of each container image
It looks for updated versions of the container image in their upstream repositories
Based on a configurable schedule, it pulls a new container image and restarts the container for updates

I encourage you to read more about watchtower on GitHub. There’s plenty you can configure, including update intervals, how updates are handled, and how you can get notifications when an update happens.

My new deployments always need watchtower running. Luckily, we can combine Fedora CoreOS’ initial provisioning system, called ignition, with podman’s new quadlet feature and launch watchtower automatically on the first boot.

Quadlets #

So what’s a quadlet?

The blog post explains it well by making containers more declarative via a familiar systemd syntax. Here’s an example .container file from the post:

[Unit]
Description=The sleep container
After=local-fs.target

[Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

You can toss this into $HOME/.config/containers/systemd/mysleep.container for rootless user containers or in /etc/containers/systemd/mysleep.container for a container running as root.

Configure a quadlet on boot #

As I mentioned earlier, I want a watchtower container running on my CoreOS nodes at first boot. Let’s start with a fairly basic butane file:

variant: fcos
version: 1.4.0
passwd:
 users:
 - name: major
 groups:
 - wheel
 - sudo
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0
storage:
 files:
 - path: /etc/containers/systemd/watchtower.container
 contents:
 inline: |
 [Unit]
 Description=Watchtower container updater
 Wants=network-online.target
 After=network-online.target

 [Container]
 ContainerName=watchtower
 Image=ghcr.io/containrrr/watchtower:1.5.3@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca
 Environment=WATCHTOWER_CLEANUP=true
 Environment=WATCHTOWER_POLL_INTERVAL=3600
 Volume=/var/run/docker.sock:/var/run/docker.sock
 SecurityLabelDisable=true

 [Install]
 WantedBy=multi-user.target default.target

Let’s break this file down:

I start by adding a user named major that has administrative privileges an an ssh key (this is optional, but I like using my own username rather than core)
The quadlet unit file lands in /etc/containers/systemd/watchtower.container and starts at boot time

The quadlet file has some important configurations:

I added environment variables to clean up outdated container images and check for updates once an hour
The podman socket is mounted inside the watchtower container
Security labels are disabled to allow for communication with the podman socket

Mounting the podman socket and disabling security labels is not an ideal security approach. However, I’ve found that watchtower’s configuration and automation fits my needs really well and I retreive the image from a trusted source. If this won’t work for you, you can use podman’s built-in auto-update feature instead.

From here, we convert the butane configuration into an ignition configuration. I’m launching this CoreOS node on VULTR, so I’ve named my files accordingly:

$ butane vultr-coreos.butane > vultr-coreos.ign

Let’s go 🚀 #

I’m using VULTR’s CLI here in Fedora, but you can do the same steps via VULTR’s portal if needed. Just paste in the ignition configuration into the large text box before launch.

# Install vultr-cli in Fedora
sudo dnf install vultr-cli

# Launch the instance
vultr-cli instance create --region dfw --plan vhp-2c-2gb-amd \
 --os 391 --label coreos-dfw-1 --host coreos-dfw-1 \
 --userdata "$(cat vultr-coreos.ign)"

Let’s see how the container is doing:

$ ssh major@COREOS_HOST

Fedora CoreOS 38.20230430.3.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[major@coreos-dfw-1 ~]$ sudo podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0024712c95d ghcr.io/containrrr/watchtower@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca About a minute ago Up About a minute watchtower

[major@coreos-dfw-1 ~]$ sudo podman logs watchtower
time="2023-05-31T14:01:12Z" level=info msg="Watchtower 1.5.3"
time="2023-05-31T14:01:12Z" level=info msg="Using no notifications"
time="2023-05-31T14:01:12Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2023-05-31T14:01:12Z" level=info msg="Scheduling first run: 2023-05-31 15:01:12 +0000 UTC"
time="2023-05-31T14:01:12Z" level=info msg="Note that the first check will be performed in 59 minutes, 59 seconds"

Awesome! 🥳

My system rebooted for an ostree update shortly after provisioning and the container came up automatically both times.

CoreOS as a pet

major@mhtx.net (Major Hayden) — Thu, 25 May 2023 00:00:00 +0000

Anyone working with containers has likely heard of CoreOS by this point Haven’t heard about it? Don’t despair. I’ll catch you up on what you missed.

Fedora CoreOS offers a really fast pathway to running containers on hardware, in virtual machines, or in clouds. It delivers a lightweight operating system with all of the container technology that you need for running simple containers or launching a kubernetes deployment.

But that’s not the best part.

CoreOS really shines due to its immutable OS layer¹. The OS underneath your containers ships as a single unit and it automatically updates itself much like your mobile phone. An update rolls down, CoreOS sets it up as a secondary OS, reboots into that new update, and rolls back to the original one if there were any issues.

Many people use CoreOS as the workhorse underneath kubernetes. Red Hat uses it underneath OpenShift as well. It’s even supported by the super light weight kubernetes distribution k3s.

But can you use CoreOS as a pet type instance that you use and maintain for long periods of time just like any other server? Absolutely!

What’s this pet stuff about? #

Whether you like it or not, there’s a cattle versus pets paradigm that took hold in the world of IT at some point. The basic ideas are these:

When you take care of cattle, you take care of them as a group. Losing one or more of them would make you sad, but you know you have many others.
As for pets, you spend a lot of time taking care of them and playing with them. If you lost one, it would be devastating.

A fleet of web servers could be treated like cattle. Keep lots of them online and replace any instances that have issues.

On the other hand, databases or tier zero systems (everyone feels if it they went down) are like pets. You carefully build, maintain, and monitor these.

How does CoreOS fit in? #

Many people do use CoreOS as a container hosting platform as part of a bigger system. It works really well for that. But it’s great as a regular cloud server, too.

You can run a single node CoreOS deployment and manage containers via the tools that you know and love. For example, docker-compose works great on CoreOS. I even used it to host my own Mastodon deployment.

You can also load up more user-friendly tools such as portainer to manage containers in a browser.

My development tools are missing! #

😱 No vim? This is too minimal! What are we going to do?

Luckily CoreOS comes with toolbox. 🧰

Toolbox gives you the ability to run a utility container on the system with some handy benefits:

Toolbox environments have seamless access to the user’s home directory, the Wayland and X11 sockets, networking (including Avahi), removable devices (like USB sticks), systemd journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc..

This means that the toolbox feels like a second OS on the system and it has all of the elevated privileges that you need to do your work. Simply run toolbox enter, follow the prompts, and you’ll end up with a Fedora toolbox that matches your CoreOS version. Need a different version, such as Fedora Rawhide? Just specify the Fedora release you want on the prompt:

$ toolbox enter --release 39 
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:39 (500MB)? [y/N]: y

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

⬢[major@toolbox ~]$

Look at the toolbox create --help output to see how to create lots of different toolbox containers with different names and releases. If you go overboard and need to delete some toolboxes, just list your toolboxes with toolbox list and follow it up with toolbox rm.

My tool won’t work in the toolbox. #

Some applications have issues running inside a container, even one that has elevated privileges on the system. CoreOS offers an option for layering packages on top of the underlying immutable OS.

Simply run rpm-ostree install PACKAGE to layer a package on top of the OS. When rpm-ostree runs, it creates a new layer and sets that layer to be active on the next boot. That means that you need to reboot before you can use the package.

Don’t want to reboot? There’s another option, but I recommend against it if you can avoid it².

You can apply a package layer live on the system without a reboot with the --apply-live flag. Installing a package like mtr would look like this:

$ sudo rpm-ostree install --apply-live mtr

As soon as rpm-ostree finishes its work, mtr should be available on the system for you to use.

How do updates work? #

There are two main technologies at work here.

First, zincati checks for updates to your immutable OS tree. It runs on a configurable schedule that you can adjust based on your preferences.

Second, rpm-ostree handles the OS layers and switches between them at boot time. If you’re running off layer A and an update comes down (layer B), that layer is written to the disk and activated on the next boot. Should there be any issues booting up layer B later, rpm-ostree switches the system back to layer A. In these situations, your downtime might be extended a bit due to two reboots. Your system will come back up with the original OS layer activated.

You also get a choice of update streams. Want to live a bit more on the edge? Go for next or testing. You’re on the stable stream by default.

Although I haven’t landed in this situation, it’s possible that the system boots into a new update where you notice a problem that doesn’t affect the boot. You can manually roll back to fix it.

I have more questions. #

Your first stop should be the Fedora CoreOS docs. There are also lots of ways to contact the development team and talk with the community.

Love the idea of an immutable OS but you wish you had it for your desktop or laptop? Go check out Fedora Silverblue. 💻

Okay, so it’s mostly immutable. You can edit configuration in /etc and you can layer more packages on top of the base OS layer if you need them. However, CoreOS maintainers discourage adding layered packages if you can avoid it. ↩︎
When you apply some packages and make them available immediately, you may lose track of which ones were applied live and which ones are available on the next reboot. Things can get a bit confusing if you suddenly change your mind about applying a package live or not. ↩︎

My beef with mailing lists

major@mhtx.net (Major Hayden) — Wed, 10 May 2023 00:00:00 +0000

🥵 This post is long. If you need a TL;DR, just hop down to the end.

One of my toots fell into a Fedora Development mailing list discussion recently that was titled “It’s time to transform the Fedora devel list into something new.” As you might imagine, that post blew up.

Here’s the toot:

After 20 days and 218 emails from 63 participants, the discussion continues. A few people reached out with questions and comments to better understand where I was coming from in that tweet.

Long story short: My beef¹ isn’t about the venue, the technology, or the people.

My issues are centered around the discourse itself and the time required to parse it.

Time #

One of my favorite leaders of all time once caught me off guard during a development conversation:

Them: Major, what’s the most valuable thing you have that you can give to someone else?

Me: What I know?

Them: No.

Me: My ability to understand what they need?

Them: No.

Me: Okay, just tell me.

Them: Your time.

He made a point that time is something you can’t get back and it’s why companies pay people. Companies pay people for their time.

Time spent doing hard things.

Time spent away from family.

Time spent building something new or repairing something that’s broken.

Time spent doing tasks that nobody else wants to do.

Humans jump at anything that saves them a little time. We stream Netflix instead of going to the video store or getting DVDs in the mail. We sign up for Amazon Prime to save time on shopping. We rely on appliances to do work for us so we have time for other things.

Mailing lists #

Anyone working on open source projects of any scale have likely used mailing lists from time to time. If you’re not familiar, here’s an example workflow:

You write an email to a special email address with a question, comment, or a request for help
That email goes into a system which distributes the email to people who are interested in that topic
Mailing list subscribers submit their replies asynchronously until the issue is resolved (or everyone is exhausted)
Hopefully you got what you needed

But this post isn’t about technology. You can have asynchronous discussions of varying quality levels just like these in other systems, such as Discourse, forums, Reddit, or bug trackers. There’s nothing inherently bad about mailing lists in general. Mailing lists just happen to be very common in open source projects.

So what is this post about?

Unorganized discourse #

When someone asks for comments on a topic, especially a controversial one, they get a wide array of replies:

People who don’t understand and have questions
People who dislike change in all its forms
People who dislike your change and provide use cases describing how it’s bad
People who dislike your change
People who heard there’s a fight and they can’t stay away
People who like parts of your idea but want to make changes
People who like everything you said²
People who meant to reply to another thread but replied on yours instead
People who are upset about someone else who top posted 🤭
No reply at all

Filtering through all of these to get to the heart of the argument is incredibly tedious and time consuming. 🥵

As Matthew Miller mentioned in his Fedora thread, long threads often cause people to lose the meaning of the discussion. They reply on third, fourth, and fifth level comments in the thread that often veer into other topics. Nobody can rein in the discussion at that point, but people do try. Again, these issues crop up in all kinds of discussion technology systems. They are not unique to mailing lists.

Improvements #

The problem is not the mailing list. It’s the discourse.

Keeping the discourse more organized is a good option, but its effectiveness is largely governed by the project’s communication rules and the civility of those who communicate in the thread. How do we make this better?

Provide the why #

I’m often amazed at some of the change proposals I see where the technical solution looks so elegant. It’s so easy to maintain. It won’t be difficult to test. We could implement it so quickly. This person is a genius!

Then I stop. Wait a minute. Why do we need to do this?

Always include some of the backstory and use cases behind the change. Ask yourself a few questions:

Why do we need to make the change?
Who benefits?
Who is harmed?
What happens if we don’t make the change?
Do we have alternatives to the proposed change?
Can the change be broken up into smaller pieces?

Include these in your original post to ensure everyone is on a level playing field at the start. This reduces replies requesting more information or questioning the value of the change based on a lack of understanding.

Intermission summary #

When I’ve written posts that blew up, I took time to read through the replies and summarize the comments thus far. I do this as a reply to my original post³ with a bulleted list.

As an example:

Thanks for all the replies! So far, this is what I've heard
in the thread:

1. Most people think the first change makes sense and should
 be done soon
2. The second change requires some more thought based on the
 use cases provided by lumpynuggets25
3. We need better documentation to explain why we are making
 this change

This saves time for newcomers to the thread since they can get a brief summary of the comments made thus far without needing to read all of them first. In addition, it gives some of the people who replied a chance to say “Yes, that’s what I meant. Thank you.” It also redirects the conversation back to the original topic and reduces the veering off into other topics.

Be patient #

I’ve seen many threads where the original author obviously felt the need to reply to every comment that came through. This lengthens the thread unnecessarily and causes you to think more about replying rather than understanding the replies. (Other participants in the thread may start doing this themselves.)

When you send something and the replies start rolling in, just be patient. Sometimes others will engage each other in the thread and answer your questions for you. This gives you time to read and understand what people are saying. You also get the opportunity to build that intermission summary from the previous section. 😉

Let it go #

Just like Elsa sang in Frozen, sometimes you have to let it go:

Let it go, let it go

Can’t hold it back anymore

Let it go, let it go

Turn away and slam the door

I don’t care what they’re going to say

Let the storm rage on

The cold never bothered me anyway

There’s always going to be that time where you need to walk away. Take the feedback you received, build on it, and deliver something valuable for people. Let the people who want to be angry for the sake of being angry just be angry on their own.

Every thread reaches that point where the people with real comments have provided all of their feedback. Other people just get exhausted with the conversation. That leaves the people who have nothing better to do, and of course, the trolls. (Don’t feed the trolls.)

TL;DR #

Just to summarize:

Mailing lists aren’t evil, but unorganized discourse in any medium becomes a time sink and that’s evil
Always provide the “why” along with the “what”
Take time to repeat back what was said in a summary
Let the comments roll in before considering a limited amount of replies
Know when to let it go and build something great

The word beef is used informally in English as a replacement for complaint. ↩︎
Always compliment these people. When people fully agree, they don’t reply that often because they feel like they have nothing to add. Sticking your neck out to say “This is good and here’s why…” is just as treacherous as disagreeing. ↩︎
Some people frown on replying to yourself in a mailing list thread. Are you adding value to the thread? If so, ignore them. ↩︎

Major Hayden

Jellyfin fatal player error

Checking the logs #

Checking the browser again #

Jellyfin deployment #

Users and groups #

Redirect local ports with firewalld

Old-school iptables methods #

Why consider firewalld? #

Port redirections in firewalld #

Extra credit #

amazon-ec2-utils in Fedora

The problem #

udev rules to the rescue #

Fix big cursors in Java applications in Wayland

Fixing big cursors #

Desktop file #

systemd does everything 😆 #

cloud-init and dhcpcd

What’s new with dhcpcd? #

But I use NetworkManager #

But I use systemd-networkd #

How can I get it right now? #

Texas Linux Fest 2024 recap 🤠

Containers talk #

Tech career talk #

Other talks #

Around the exhibit hall #

Roll your own static blog analytics

Why do you need analytics? #

What are the ingredients? #

A static blog in a container? #

Caddy logs #

Enabling analytics #

Serving the blog #

What about new posts? #

Connect Caddy to Porkbun

DNS validation #

Adding Porkbun support to Caddy #

Automated Caddy builds with updates #

Connecting to Porkbun #

Renewals #

Further reading #

Linux on the AMD ThinkPad Z13 G2

Power management #

Touchpad #

Display #

Audio #

Everything else #

Dark mode in Sway

GTK applications #

QT applications #

Alternate dark mode based on time #

On diversity

My first experience #

Getting underway #

Different view #

Diversity challenges #

In the eye of the beholder #

Rubber meets the road #

Another viewpoint #

My goals #

Horror book reviews from October 2023

The Cabin at the End of the World #

My thoughts #

The Ruins #

My thoughts #

The Troop #

My thoughts #

Devolution: A Firsthand Account of the Rainier Sasquatch Massacre #

My thoughts #

Tender is the Flesh #

My thoughts #

What’s next? #

Moving to cloud is more than just a purchasing exercise

Cloud offers a chance to start over #

Cloud offers a chance to software-define (nearly) anything #

Cloud offers managed services #

Cloud offers geographic distribution #

Clouds offer purchasing efficiency #