Major Hayden

Repairing 4Runner skid plate bolts

major@mhtx.net (Major Hayden) — Tue, 08 Oct 2024 00:00:00 +0000

I replaced my old Toyota 4Runner with a new one so I could snag the last run of the 5th generation. It’s a tough, reliable vehicle with just enough space for my family and our pets.

However, this new one came with the same problem as my old one. All of the bolts that hold in the front skid plate were mostly stripped. Getting them out was difficult and I knew getting them back in would be worse.

I’ll cover how to fix it in this post. If you’re in a hurry, scroll past the next section! Otherwise, let’s get a little backstory.

Backstory #

All 4Runner models are made in Japan in the Tahara Plant. The build quality is fantastic and you can tell that they’re built with care. You can even find brief notes in Japanese inside the fenders or underneath the car where someone jotted some notes about something.

After careful assembly in Japan, they head to various US ports where American workers add on any extra items that come with the trim level. That could include an upgraded exhaust, different wheels, or in my case, skid plates.

The skid plate is a sturdy piece of steel that mounts under the front of the vehicle and protects lots of important components from damage. You can see it under the front bumper in this photo:

Skid plate on a white 4Runner

If you’ve ever owned a Toyota, you know that the factory is strict about torque applied to various bolts all over the vehicle. All of that gets thrown out the window when the workers at the US ports add on accessories.

Based on all the complaints I’ve seen across various 4Runner forums, they must use air wrenches or some kind of impact wrench to put on the bolts. If the bolt isn’t in straight when they start, it destroys the bolt and causes problems with the mount holes. They also tighten the bolts far past the acceptable torque specs.

To make matters worse, if you take your car in for an oil change at most places, they’ll have an impact wrench handy to ruin the bolts a bit more for you.

Root cause #

If you have bolts that are getting stripped in the mount holes or they’re getting stuck as you try to bring the bolts in or out, you likely have chunks of metal from the bolts wedged in the threads of the bolt holes.

Ingredients #

The fix is quite cheap but very tedious. You’ll need a few parts to get started:

Irwin Hanson 12002 T-Handle Tap Wrench (1/4" to 1/2"): This T-Handle wrench allows you to easily spin the tap screw to clear the metal fragments from your bolt holes. Don’t use a socket set, drill, screwdriver, or anything powerful! You want to take this slow.
Irwin Tap 10-1 25mm Plug: The bolts that go into the holes are M10 bolts with a 25mm thread, so this tap should fit perfectly.
Toyota part PT938-00140-AA: This includes four new bolts with spacers and retaining washers to replace your stripped bolts.
14mm socket and socket wrench OR a 14mm wrench: You’ll need this for removing the bolts and dealing with the hardware for the skid plate.
Some type of lubricant. I used WD-40, but don’t tell anyone. People love to fight about whether WD-40 is a solvent, a grease, or a lubricant. 🤷‍♂️

What’s hilarious is that if you load the Amazon page for the Toyota part, it shows that everyone is buying tap screws and T-Handle wrenches:

Even Amazon knows these bolts are a problem! 😆

Fix the bolt holes #

First things first, you’ll need to get that skid plate off. I strongly recommend taking the rear bolts off first. If the bolts get stuck on the way out, take your time. I found that rocking in the other direction briefly and then trying to loosen them again seemed to work.

With the rear bolts out, move to the bolts closest to the front of the car. Put a box or something sturdy underneath the skid plate that allows it to drop when it’s loose but prevents it from knocking out one of your teeth when it falls. 🤕

When you loosen the front bolts, try loosening one of them 4-5 turns and then go to the other one. Keep going back and forth loosening the bolts until they loosen from the frame of the car. There are retaining washers on the top side of the bolt and removing those bolts aggressively will slide the retaining washers right off the bolt.

Now you’re ready to tap! 👏

Start in with the bolt holes in the rear and spray a decent amount of lubricant in the bottom and top of the bolt hole. Get your tap into the T-Handle wrench and slowly start turning it in the bolt hole like you were installing one of the bolts.

🛑 When you hit resistance, only go 1/2 to 1 turn further. Then back up 2-3 turns. This means you’ve dislodged some metal fragments in the threads!

Working on one of the back holes 💪

After backing up a bit, keep screwing it in further until you hit more resistance. Only go 1/2 to 1 turn more, then back out 2-3 turns. Keep doing this until your tap shows up out of the top side of the bolt hole.

With your tap sticking out of the top of the hole, grab a shop towel or paper towel and clear all of the metal filings away from the top of the hole. Then back the tap all the way out and clean your tap screw. It’s likely going to be covered in black shavings.

If you want to be really thorough, lubricate the hole once more and keep working the tap until the threads feel really smooth. I added some lubricant and fed a new bolt in through the top using finger strength only until I knew the threads were clear.

You can test feed a new bolt through the bottom to verify that you’ve done a good job. Don’t use the old bolts for this. You’ll just get more junk in the threads again. 😭

Repeat these steps for the other four holes and you should be all set.

Replace the skid plate #

Be sure to discard the old bolts to avoid causing more problems for yourself. Re-assemble the new front bolts with the spacer and retaining washer just like they were when you took the skid plate down.

Get the back bolts going in first and get them in about halfway. Start working on the front bolts after that.

When all four bolts are in, grab your torque wrench and tighten them to 22 ft/lbs or 30 Nm as specified in the manual:

Always check the instructions for torque specs! 🔧

If you don’t own a torque wrench, I wouldn’t tighten them much past finger tight with a socket wrench. Then drive to the hardware store and get a basic torque wrench. 😜

Prevention #

This is easy but annoying: always remove the skid plate yourself before any kind of maintenance trip.

Yes, this sounds silly, but these mount points are finicky and there’s no guarantee that the dreaded impact wrench will not show up again to ruin your bolts. I remove mine before any oil changes or scheduled maintenance at the dealer.

Dealers have asked me in the past “Where’s your skid plate anyway?” and I let them know I don’t want my bolts stripped. 😅

Spell check in multiple languages with Firefox

major@mhtx.net (Major Hayden) — Sun, 25 Aug 2024 00:00:00 +0000

Bienvenidos!¹ I’ve been learning Spanish for just over a year and I often type messages in either Spanish or English (my native language) with coworkers and friends. Just like most people, I make spelling mistakes in both languages. 🙃

Firefox offers a feature for multi-language spell checking and translations but it can be a bit challenging to set up. This post explains how to load languages into Firefox and use them for spell checking.

Installing languages #

Take a trip over to [Dictionaries and Languages Packs] on Mozilla’s site. Note that there are two columns available to you here:

Language packs give you the option to change your interface language to something different than your system’s default language.
Dictionaries help with checking spelling.

In the second column, click on the language you want to add for checking spelling. In my case, I picked the Spanish (Spain) Dictionary along with the Spanish (Mexico) Dictionary. Install the dictionaries you want just like any other add-on!

Go to the about:addons page in Firefox and you should see your languages under Languages and Dictionaries on the left side.

Enable the language #

Find an input field and right click inside the field. You should see a Languages context menu appear. Roll over that menu and a new menu pops out to the side:

Firefox context menu showing multiple languages

Click the checkbox to enable the languages that you want to use with the spell checker. That takes effect immediately!

Gracias por leer hasta aquí!² 😜

Welcome! ↩︎
Thank you for reading this far. ↩︎

My meeting hacks

major@mhtx.net (Major Hayden) — Thu, 22 Aug 2024 00:00:00 +0000

Ask anyone about the toughest part of their workday and it usually comes down to one thing: meetings. There are plenty of reasons:

The meeting could have been an email
Nobody notices when I attend the meeting, but they notice when I don’t
The meeting is recurring whether there’s something important to talk about or not
There’s no time for questions after everyone presents in a meeting
Someone dominates the conversation

This was a central problem in my “Five tips for a thriving technology career” talk that I delivered this year. I wrote a recap on the blog earlier this summer as well.

I came up with some more ideas since then, so let’s go!

Use headphones or earbuds #

I find it much easier to understand people in meetings when I have the audio closer to my ears. This helps a lot with understanding non-native English speakers or some native English speakers with thick accents. It reduces the noise from various things in my house (kids, pets, appliances) and allows me to focus on the small sounds that are important for understanding someone else.

How many times have you been in a meeting with someone who talks constantly without earbuds or headphones and you can’t break through with your own voice? Some computers will mute the incoming audio to avoid feedback sounds and you’ll totally miss it when someone is trying to get your attention.

I was once in a meeting where an attendee spoke at length about a topic that was already covered and multiple people tried to speak to let him know that he could stop. He was completely oblivious. The situation improved a lot recently with the addition of “raised hands” indicators in most meeting applications, but it’s still not perfect.

Background music #

If you typically join meetings without earbuds or headphones, then this suggestion isn’t for you. Also, you should go back and re-read the previous section. ️😜

Everyone has their own music preferences, but I find that playing some relaxing music at a low volume really helps me stay focused during meetings. I change the genre of music between different days depending on my mood. No matter what you choose, consider music without vocals to avoid distractions.

A good place to start is Lofi Girl’s “beats to relax/study to” playlist. You can listen on Spotify or on YouTube. Very few songs have vocals, and if they do, it’s barely noticeable. I’ve found that I can keep this playlist on for hours without getting bored of it.

This can be especially helpful for those marathon half or full day meetings. 👔

Ask about taking notes #

Most meeting platforms offer transcription and audio/video recording already, but transcriptions are difficult to read and recordings aren’t usually fun to watch. I love it when someone takes some concise notes about the points that were raised, who raised them, and who holds the action items to solve them.

If nobody’s taking notes, ask if you can!

It’s a great way to ensure you pay attention and the people who missed the meeting will thank you later. Nobody has turned me down yet when I’ve asked.

This can also be helpful if someone likes to talk over everyone else during the meeting or if someone birdwalks into other topics. Un-mute yourself and ask:

Wait, are we still on that previous topic or have we moved to something else?

Another favorite question of mine is:

Did we get an action item for that previous topic? Who owns that item?

Your note taking keeps speakers on track and ensures there is accountability and ownership for problems that need to be solved. It’s also a great way to get your name in front of other people during larger meetings.

Decline that meeting! #

This one got the biggest reaction during my talk at Texas Linux Fest. Sometimes you just need to decline a meeting. Certain aspects of a meeting will push me to the “No” button faster than others, but here’s my two biggest red flags:

More than three attendees: It’s difficult to get much done with a meeting that has 25 people in it. If someone sends me a calendar invitation unannounced and there are more than 3-5 people in the meeting, I ask them on Slack what I’m expected to bring to the meeting. Often times, I hear “Oh, we wanted to be sure you were informed, but there are no action items for you.” That’s a great time to say: “Can you send me the recording or the notes when it’s over?”
Missing agenda: If I’m taking time out of my day to meet, I want to know about the meeting’s goals. What should we have as we leave the meeting? Will we leave with a plan to do something? A set of decisions? Questions for another team?

You are the only one that can advocate for your own time. Nobody else is going to do that for you¹. A very talented executive once told me:

Time is the most valuable thing you bring to work every day. You can’t get more of it, but you can waste it. Your experience and knowledge means nothing if you don’t have time to use it. Treat your time as your most precious asset.

An administrative assistant can help but I’ve never had one before. 😜 ↩︎

Rub some AI on it

major@mhtx.net (Major Hayden) — Wed, 21 Aug 2024 00:00:00 +0000

Author’s note: This post is all about my personal thoughts on artificial intelligence (AI) and they don’t represent the views of any employer or group.

You can’t escape the clutches of AI lately.

It’s in my smartphone nestled next to my text messages. It’s in my work chats. It’s reading my Spanish in Duolingo. It’s in my photo albums retouching my images.

Sometimes we know that there’s AI involved in something and sometimes we don’t.

However, it seems like so many are in a rush to implement some kind of AI offering without a full idea of why they’re doing it. Here an excerpt from the September 2024 issue of Harvard Business Review that explains it well:

Smart early movers in sectors adopting gen AI have certainly captured some of this value in the short term. But relatively soon all surviving companies in those sectors will have applied gen AI, and it won’t be a source of competitive advantage for any one of them, even where its impact on business and business practices will probably be profound. In fact, it will be more likely to remove a competitive advantage than to confer one. But here’s a silver lining: If you already have a competitive advantage that rivals cannot replicate using AI, the technology may serve to amplify the value you derive from that advantage.

AI can help you only if:

You have a product or service your customers value.
You can leverage AI for specific improvements to that product or service that make it more valuable.

AI is not valuable alone #

I’m reminded of a time in the past where I was working hard on OpenStack public clouds. Kubernetes gained more traction day by day. Lots of customers told us: “I’ve got to get on kubernetes so I can move faster.”

As we asked more about their challenges, they listed lots of things that should look familiar:

Developers throwing code over the wall to Q/E and Q/E delays the release
Software passes tests in development and staging, but fails miserably in production
Monolithic applications were crushed by load spikes
Operations teams struggled to deploy software efficiently and reliably

They had a serious problem with delivering their software, but kubernetes couldn’t make any of these better. Adding kubernetes would just give them two problems instead of one.

The running joke whenever someone ran into a problem with a server, a piece of code, or a service was to say “Why don’t you rub a little kubernetes on it?” 🤣

I’m seeing much of the same with AI as companies scramble to get their hands on the best hardware they can find and access to the highest quality large language models (LLMs) they can find. Cloud budgets are blown wide open. When someone asks about the AI effort, the reply is often: “We have to get it before our competitors do, or we’re sunk!”

In February 2024, 36% of company earnings reports mentioned AI – a record high:

How many of them are actually doing something meaningful for their employees or customers with AI?

Work backwards from the experience #

One of my coworkers, Scott McCarty, wrote a great post on InfoWorld titled “What generative AI can do for sysadmins”. What I love most about this article is that Scott remains laser-focused on the experiences and challenges that AI could improve.

There are plenty of challenging situations that every sysadmin faces. The worst of these are when you’re under incredible pressure to bring a system back into a working state and you need to pick through tons of information to identify the problem. You can sometimes spot these issues easily, such as a failing storage drive in a server. Other situations are much more difficult.

The key is to start with the experience. Then work backwards from there.

As an example, one pattern I often see is companies putting AI chatbots in front of their documentation. Sometimes the chatbot will help you find the right documentation faster, but sometimes it’s not much better than a CTRL-F or a quick look at the documentation’s table of contents.

If your documentation is so complicated that you need to spend the time and money to put an AI chatbot in front of it, why not make your documentation better instead?

When something does fail, why not put a link to the documentation in the log message itself? This pattern shows up a lot in modern software lately. If I try to enable a Tailscale exit node but I haven’t forwarded packets on an interface, I get quick instructions on the console with a link to documentation that explains it in more detail.

You cannot use AI to paper over a poor experience. Your customers will see right through it.

Remember the human side #

Sometimes companies simply try to take AI much too far and upset the human nature in all of us.

A great example of this was Google’s awful Olympics ad where it shows a girl’s father using Google Gemini to write a letter to her hero. The reaction at my house when we saw it was: “Wait, you’re taking the time to write a letter to your hero and you’re letting an AI write it? Could that be any more impersonal?” If I’m writing a letter or email to someone I admire, I’m taking the time to write it myself with my own voice.

Another example is a Microsoft ad showing someone turning a long document into a long slide deck instead. If nobody wanted to read the document in the first place, why would they want to read your long slide deck? Also, how would they feel if they know you just jammed a document into a LLM to make a slide deck and then held them hostage in a conference room as you walked through a voiceless set of slides?

This goes back to the last section, but if you’re trying to add AI to replace a human interaction, think that through. Are you papering over a bad experience? Are you looking to cut costs without considering the customer reaction? What’s your plan if the AI interactions backfire?

So what do we do? #

If you work backwards from the customer experience and land on an LLM as the best way to solve a problem or enhance a product, that’s great. However, the experience you are enabling should be so good that:

Customers are genuinely delighted with the experience without knowing AI is involved
You don’t have to mention “AI” for the experience to feel innovative and delightful
You have plans in place for when customers want more from the experience later

Getting hardware or cloud infrastructure together and running an LLM on top is boring. Even going retrieval augmented generation (RAG) is boring. We will soon live in a world where running an LLM is the same level of difficulty as running a web server or a container. That’s not where the value lives.

AI isn’t the king. The experience is. If you forget that, you’re just taking a problem and rubbing some AI on it.

AMD GPU missing from btop

major@mhtx.net (Major Hayden) — Tue, 20 Aug 2024 00:00:00 +0000

I recently built a new PC for my birthday and I splurged a bit with a new AMD Radeon 7900 XTX GPU. Although I’m not a heavy gamer, I’m working with LLMs more often and I’m interested to do some of this work at home.

btop is my go-to tracker for all kinds of data about my system, including CPU usage, memory usage, disk I/O, and network throughput. It’s a great way to track down bottlenecks and find out why your CPU fan is spinning at max speed. 😜

Screenshot of btop running on my system

The problem #

My GPU wasn’t showing up in btop after rebuilding the system. Normally, there’s a bar for the GPU right underneath the CPU usage and it tracks the GPU usage as well as memory usage. Some cards report thermals there, too.

The radeontop tool worked fine and I can see the device in the hardware monitoring subsystem:

> cat /sys/class/hwmon/hwmon1/name
amdgpu

The solution #

I installed plenty of AMD packages, but I missed a critical one: rocm-smi:

> dnf install rocm-smi
> rocm-smi

======================================== ROCm System Management Interface ========================================
================================================== Concise Info ==================================================
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 
 (DID, GUID) (Edge) (Avg) (Mem, Compute, ID) 
==================================================================================================================
0 1 0x744c, 55924 47.0°C 21.0W N/A, N/A, 0 218Mhz 96Mhz 0% auto 327.0W 11% 10% 
==================================================================================================================
============================================== End of ROCm SMI Log ===============================================

That’s the ticket! Now my btop data is complete.

btop showing my GPU stats

Running ollama with an AMD Radeon 6600 XT

major@mhtx.net (Major Hayden) — Thu, 08 Aug 2024 00:00:00 +0000

I’m splitting time between two roles at work now and one of the roles has a heavy focus on LLMs. Much like many of you, I’ve given ChatGPT a try with questions from time to time. I’ve also used GitHub Copilot within Visual Studio Code.

They’re all great, but I was really hoping to run something locally on my machine at home.

Then I stumbled upon a great post on All Things Open titled “Build a local AI co-pilot using IBM Granite Code, Ollama, and Continue” that started me down a path with ollama. The ollama project gets you started with a local LLM and makes it easy to serve it for other applications to use.

It’s so slow 🐌 #

When I first began connecting vscode to ollama, I noticed that the responses were incredibly slow. A quick check with btop showed that my CPU was maxed out at 100% utilization and my GPU was entirely idle. That’s not good.

My first thought was to check the system journal with sudo journalctl --boot -u ollama. That gets me all the messages from ollama since I last booted the machine.

source=images.go:781 msg="total blobs: 0"
source=images.go:788 msg="total unused blobs removed: 0"
source=routes.go:1155 msg="Listening on 127.0.0.1:11434 (version 0.3.4)"
source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1586759388/runners
source=payload.go:44 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cuda_v11 rocm_v60102 cpu]"
source=gpu.go:204 msg="looking for compatible GPUs"
source=amd_linux.go:59 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
source=amd_linux.go:340 msg="amdgpu is not supported" gpu=0 gpu_type=gfx1032 library=/usr/lib64 supported_types="[gfx1030 gfx1100 gfx1101 gfx1102]"
source=amd_linux.go:342 msg="See https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides for HSA_OVERRIDE_GFX_VERSION usage"
source=amd_linux.go:360 msg="no compatible amdgpu devices detected"

A couple of things in the output stood out to me:

stat /sys/module/amdgpu/version: no such file or directory
msg="amdgpu is not supported" gpu=0 gpu_type=gfx1032 library=/usr/lib64 supported_types="[gfx1030 gfx1100 gfx1101 gfx1102]"
"See https://github.com/ollama/ollama/blob/main/docs/gpu.md#overrides for HSA_OVERRIDE_GFX_VERSION usage"

Sure enough, the version was missing:

> stat /sys/module/amdgpu/version
stat: cannot statx '/sys/module/amdgpu/version': No such file or directory

And my AMD GPU is indeed an AMD Navi 23 chipset (gfx1032):

> lspci | grep -i VGA
0f:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c7)

I went over to the linked overrides documentation to figure out what HSA_OVERRIDE_GFX_VERSION is all about:

Ollama leverages the AMD ROCm library, which does not support all AMD GPUs. In some cases you can force the system to try to use a similar LLVM target that is close. For example The Radeon RX 5400 is gfx1034 (also known as 10.3.4) however, ROCm does not currently support this target. The closest support is gfx1030. You can use the environment variable HSA_OVERRIDE_GFX_VERSION with x.y.z syntax. So for example, to force the system to run on the RX 5400, you would set HSA_OVERRIDE_GFX_VERSION=“10.3.0” as an environment variable for the server. If you have an unsupported AMD GPU you can experiment using the list of supported types below.

The fix #

The docs recommended setting HSA_OVERRIDE_GFX_VERSION="10.3.0" to see if my card will work. Let’s edit the systemd unit file for ollama to drop in some additional configuration:

> sudo systemctl edit ollama.service

An editor appeared with text in it:

### Editing /etc/systemd/system/ollama.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

### Edits below this comment will be discarded

So I added the suggested override along with the path to my AMD ROCm directory:

### Editing /etc/systemd/system/ollama.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
Environment="ROCM_PATH=/opt/rocm"

### Edits below this comment will be discarded

Then I can tell systemd to reload the unit and restart ollama:

> sudo systemctl daemon-reload
> sudo systemctl stop ollama
> sudo systemctl start ollama

Back to the system journal for another look:

source=amd_linux.go:348 msg="skipping rocm gfx compatibility check" HSA_OVERRIDE_GFX_VERSION=10.3.0
source=types.go:105 msg="inference compute" id=0 library=rocm compute=gfx1032 driver=0.0 name=1002:73ff total="8.0 GiB" available="5.9 GiB"

Success! 🎉

Giving it another try #

I went back to vscode and tried some code completions, but they were only slightly faster than using the CPU. Each time I’d wait for completion, I’d watch btop and the GPU would spike, then the CPU, then the GPU spikes again, and so on.

After talking with a coworker, it looks like my Radeon 6600 XT is great for games, but it lacks the RAM needed to load the model into the GPU. 😭 From what I’ve read, 24GB is the suggested minimum and that’s the largest amount of RAM you’ll find in most GeForce/Radeon consumer graphics cards.

Jellyfin fatal player error

major@mhtx.net (Major Hayden) — Tue, 02 Jul 2024 00:00:00 +0000

Plex has been a mainstay for serving up media at home but it seems to have changed lately towards a more and more commercial offering. A friend recommended Jellyfin and I deployed it on my Synology NAS in a Docker container.

I did a few quick tests in a web browser and everything looked good. But then my Jellyfin android app told me:

Playback failed due to a fatal player error

Everything looked fine in the browser, so it was time to dig in.

Checking the logs #

I opened up an ssh connection on the Synology to check the logs and found something unhelpful:

Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 1

Running a few searches led me down rabbit holes to plenty of GitHub issues. None of them fixed the issue.

Checking the browser again #

I went through a few different videos from the Synology and played each. They all looked fine in Firefox until I reached one that seemed to stutter. The frame rate looked as if at least half of the frames were bring dropped.

That particular video was in 4K with a high bit rate. Back on the synology, the CPU usage was through the roof.

I configured graphics acceleration when I deployed Jellyfin. Perhaps it wasn’t working?

Jellyfin deployment #

I deployed Jellyfin using the upstream guides with docker-compose:

jellyfin:
 image: docker.io/jellyfin/jellyfin:latest
 container_name: jellyfin
 user: 1026:100
 network_mode: "host"
 devices:
 - /dev/dri
 volumes:
 # removed
 restart: "unless-stopped"

One of the GitHub issues I stumbled upon suggested being specific about the video devices that are mounted inside the container.

$ ls -al /dev/dri
total 0
drwxr-xr-x 2 root root 80 Jun 10 20:23 .
drwxr-xr-x 12 root root 14140 Jun 10 20:24 ..
crw------- 1 root root 226, 0 Jun 10 20:24 card0
crw-rw---- 1 root videodriver 226, 128 Jun 10 20:24 renderD128

I adjusted the deployment in docker-compose.yaml and tried again:

jellyfin:
 image: docker.io/jellyfin/jellyfin:latest
 container_name: jellyfin
 user: 1026:100
 network_mode: "host"
 devices:
 - /dev/dri/renderD128:/dev/dri/renderD128
 - /dev/dri/card0:/dev/dri/card0
 volumes:
 # removed
 restart: "unless-stopped"

I redeployed jellyfin:

$ docker-compose up -d jellyfin

The Android app still had the fatal player error.

Users and groups #

Most of my Synology containers use the uid/gid pair of 1026:100 so allow them to read and write to my storage volume. The /dev/dri/renderD128 is owned by the videodriver group:

$ grep videodriver /etc/group
videodriver::937:PlexMediaServer

This likely came from a time when I installed Plex on Synology via one of the Synology applications rather than from a container. (I’m not sure, but that’s my guess.)

I added that group to the container:

jellyfin:
 image: docker.io/jellyfin/jellyfin:latest
 container_name: jellyfin
 user: 1026:100
 network_mode: "host"
 group_add:
 - "937"
 devices:
 - /dev/dri/renderD128:/dev/dri/renderD128
 - /dev/dri/card0:/dev/dri/card0
 volumes:
 # removed
 restart: "unless-stopped"

After redeploying the container, the Android app worked just fine! Also, the video stuttering disappeared when viewing the 4K video from the browser. 🎉

Redirect local ports with firewalld

major@mhtx.net (Major Hayden) — Fri, 28 Jun 2024 00:00:00 +0000

Linux networking and firewalls give us plenty of options for redirecting traffic from one port to another. We can allow people outside our home to reach a web server we run in our internal network. That’s called destination NAT, ot DNAT.

You can also redirect traffic to different ports on the same host. For example, if you have a daemon listening on port 3000, but you want people to reach that service on port 80, you can redirect traffic from 80 to 3000 on the same host (without network address translation).

But how do we do this with firewalld? 🤔

Old-school iptables methods #

Let’s say you have a service running on port 3000 and you want to expose it to other computers on your same network as port 80. With iptables, you would typically start by enabling IP forwarding:

sudo sysctl -w net.ipv4.ip_forward=1

Add two iptables rules to handle packets coming in from the outside as well as any locally generated packets:

# Handle locally-generated packets on the same machine.
sudo iptables -t nat -A PREROUTING -s 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to 3000`

# Handle packets coming from outside the current machine.
sudo iptables -t nat -A OUTPUT -s 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to 3000`

There’s a weird situation that happens on certain machines with certain network configurations where packets are not properly routed when they are destined for the local network adapter. To fix that, set one more sysctl configuration:

sudo sysctl -w net.ipv4.conf.all.route_localnet=1

Remember to make these sysctl configurations permanent:

sudo mkdir /etc/sysctl.conf.d/
echo "net.ipv4.ip_forward=1" | sudo tee >> /etc/sysctl.conf.d/redirect.conf
echo "net.ipv4.conf.all.route_localnet" | sudo tee >> /etc/sysctl.conf.d/redirect.conf

Why consider firewalld? #

I like firewalld because I can manage lots of settings for different firewall zones and allow access from one zone to another. It also allows me to put certain interfaces in trusted zones so they automatically get more access.

Another nice aspect about firewalld is that it supports iptables and nftables backends. You don’t have to think about the differences between the backends. All of that is taken care of for you.

Port redirections in firewalld #

Let’s start by checking our default firewalld zone:

$ sudo firewall-cmd --list-all
FedoraServer (default, active)
 target: default
 ingress-priority: 0
 egress-priority: 0
 icmp-block-inversion: no
 interfaces: bond0 eno1 eno2
 sources: 
 services: dhcpv6-client http https
 ports: 51820/udp
 protocols: 
 forward: yes
 masquerade: yes
 forward-ports:
 source-ports: 
 icmp-blocks: 
 rich rules:

This output shows that my external network interfaces are attached to the zone and forwarding is already on in my case. If you see forward: no here, just run this command:

$ sudo firewall-cmd --add-forward
success

Now firewalld will manage your forwarding sysctl variables for you on these interfaces. That’s handy. 😉

Next, let’s get the redirect working. We want to take external packets on port 80 and send them to 3000: on the local machine.

$ sudo firewall-cmd \
 --add-forward-port=port=80:proto=tcp:toport=3000:toaddr=127.0.0.1
success

In this command, we told firewalld to take 80/tcp from the outside and send it to port 3000 on the local host (127.0.0.1). Let’s double check our current configuration:

$ sudo firewall-cmd --list-all
FedoraServer (default, active)
 target: default
 ingress-priority: 0
 egress-priority: 0
 icmp-block-inversion: no
 interfaces: bond0 eno1 eno2
 sources: 
 services: dhcpv6-client http https
 ports: 51820/udp
 protocols: 
 forward: yes
 masquerade: yes
 forward-ports: 
	port=80:proto=tcp:toport=3000:toaddr=127.0.0.1
 source-ports: 
 icmp-blocks: 
 rich rules:

Test a connection to port 80 with curl and it should redirect to the service on port 3000.

🚨 If everything works, remember to save the firewalld configuration:

$ sudo firewall-cmd --runtime-to-permanent
success

Extra credit #

We can inspect the nftables rules to see the firewall rules that firewalld set for us. The Arch Linux nftables wiki page is superb for looking up those commands.

If we dump the current ruleset, we see the rule we created in firewalld:

$ sudo nft list ruleset
---SNIP---
chain nat_PRE_FedoraServer_allow {
 meta nfproto ipv4 tcp dport 80 dnat ip to 127.0.0.1:3000
}
---SNIP---

amazon-ec2-utils in Fedora

major@mhtx.net (Major Hayden) — Wed, 08 May 2024 00:00:00 +0000

We’ve all been in that situation where we see a device in Linux and wonder which physical device it corresponds to. I remember when I built my first NAS and received an alert that a drive had failed. It took me a while to figure out which physical drive actually needed to be replaced.

This happens with network devices, too, and I wrote a post about systemd’s predictable network device names back in 2015.

Cloud instances often make it even more confusing because storage devices are fully virtualized and show up differently depending on the cloud provider. I recently packaged amazon-ec2-utils in Fedora to make this a little easier on AWS.

The problem #

I just built a test instance of Fedora 40 in AWS and the AWS API shows the block device mappings like this:

$ aws ec2 describe-instances \
 --instance-ids i-0687448a184ab0a9e | \
 jq '.Reservations[0].Instances[0].BlockDeviceMappings'
[
 {
 "DeviceName": "/dev/sda1",
 "Ebs": {
 "AttachTime": "2024-05-08T15:24:03+00:00",
 "DeleteOnTermination": true,
 "Status": "attached",
 "VolumeId": "vol-0832569729b6c5ea6"
 }
 }
]

However, if I check these devices inside the instance itself, I get something totally different:

[fedora@f40 ~]$ sudo fdisk -l
Disk /dev/nvme0n1: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Amazon Elastic Block Store 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9FB58ED7-7581-4469-BEB7-64F069151EAF

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 206847 204800 100M EFI System
/dev/nvme0n1p2 206848 2254847 2048000 1000M Linux extended boot
/dev/nvme0n1p3 2254848 20971484 18716637 8.9G Linux root (ARM-64)


Disk /dev/zram0: 1.78 GiB, 1909456896 bytes, 466176 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

[fedora@f40 ~]$ ls -al /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory

One disk isn’t so bad, but what if we add more storage? The API tells me one thing:

> aws ec2 describe-instances --instance-ids i-0687448a184ab0a9e | jq '.Reservations[0].Instances[0].BlockDeviceMappings'
[
 {
 "DeviceName": "/dev/sda1",
 "Ebs": {
 "AttachTime": "2024-05-08T15:24:03+00:00",
 "DeleteOnTermination": true,
 "Status": "attached",
 "VolumeId": "vol-0832569729b6c5ea6"
 }
 },
 {
 "DeviceName": "/dev/sde",
 "Ebs": {
 "AttachTime": "2024-05-08T15:38:29.754000+00:00",
 "DeleteOnTermination": false,
 "Status": "attached",
 "VolumeId": "vol-0a7ba05c5270d7aa3",
 "VolumeOwnerId": "xxx"
 }
 }
]

But then the instance tells me something else entirely:

[fedora@f40 ~]$ sudo fdisk -l
Disk /dev/nvme0n1: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Amazon Elastic Block Store 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9FB58ED7-7581-4469-BEB7-64F069151EAF

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 206847 204800 100M EFI System
/dev/nvme0n1p2 206848 2254847 2048000 1000M Linux extended boot
/dev/nvme0n1p3 2254848 20971484 18716637 8.9G Linux root (ARM-64)


Disk /dev/zram0: 1.78 GiB, 1909456896 bytes, 466176 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/nvme1n1: 10 GiB, 10737418240 bytes, 20971520 sectors
Disk model: Amazon Elastic Block Store 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

udev rules to the rescue #

The amazon-ec2-utils package provides some helpful udev rules and scripts to make it easier to identify these devices. This package is on the way to Fedora as I write this post, but it hasn’t reached the stable repos yet. Once it does, you should be able to install it:

$ sudo dnf install amazon-ec2-utils

In the meantime, you can download the latest build and install it on your instance:

$ sudo dnf install /usr/bin/koji
$ koji download-build amazon-ec2-utils-2.2.0-2.fc40 
Downloading [1/2]: amazon-ec2-utils-2.2.0-2.fc40.src.rpm
[====================================] 100% 24.01 KiB / 24.01 KiB
Downloading [2/2]: amazon-ec2-utils-2.2.0-2.fc40.noarch.rpm
[====================================] 100% 20.53 KiB / 20.53 KiB
$ sudo dnf install amazon-ec2-utils-2.2.0-2.fc40.noarch.rpm

The cleanest method to get these new udev rules working is to reboot, but if you’re in a hurry, there’s an option to reload these rules without a reboot:

$ sudo udevadm control --reload-rules
$ sudo udevadm trigger

What do we have in /dev/ now?

[fedora@f40 ~]$ ls -al /dev/sd*
lrwxrwxrwx. 1 root root 7 May 8 15:44 /dev/sda1 -> nvme0n1
lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda11 -> nvme0n1p1
lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda12 -> nvme0n1p2
lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda13 -> nvme0n1p3
lrwxrwxrwx. 1 root root 7 May 8 15:44 /dev/sde -> nvme1n1

We can put a filesystem down on the new device using the same name as the API presents:

$ sudo dnf install /usr/sbin/mkfs.btrfs
$ sudo mkfs.btrfs /dev/sde
btrfs-progs v6.8.1
See https://btrfs.readthedocs.io for more information.

Performing full device TRIM /dev/sde (10.00GiB) ...
NOTE: several default settings have changed in version 5.15, please make sure
 this does not affect your deployments:
 - DUP for metadata (-m dup)
 - enabled no-holes (-O no-holes)
 - enabled free-space-tree (-R free-space-tree)

Label: (null)
UUID: c2fb9e33-3bf6-4b5b-aa80-44e315f499de
Node size: 16384
Sector size: 4096	(CPU page size: 4096)
Filesystem size: 10.00GiB
Block group profiles:
 Data: single 8.00MiB
 Metadata: DUP 256.00MiB
 System: DUP 8.00MiB
SSD detected: yes
Zoned device: no
Features: extref, skinny-metadata, no-holes, free-space-tree
Checksum: crc32c
Number of devices: 1
Devices:
 ID SIZE PATH 
 1 10.00GiB /dev/sde

Being able to know these device names during the instance launch or during storage operations makes it much easier to write automation for these devices. There’s no guess work required to translate the device that an instance shows you to what you see via the API.

Fix big cursors in Java applications in Wayland

major@mhtx.net (Major Hayden) — Fri, 26 Apr 2024 00:00:00 +0000

Scroll through the list of Wayland posts posts on the blog and you’ll see that I’ve solved plenty of weird problems with Wayland and the Sway compositor. Most are pretty easy to fix but some are a bit trickier.

Java applications are notoriously unpredictable and Wayland takes unpredictability to the next level. One particular application on my desktop always seems to start with massive cursors.

This post is about how I fixed and then discovered something interesting along the way.

Fixing big cursors #

I recently moved some investment and trading accounts from TD Ameritrade to Tastytrade. Both offer Java applications that make trading easier, but Tastytrade’s application always started with massive cursors.

To make matters worse, sometimes the cursor looked lined up on the screen but then the click landed on the wrong buttons in the application! Errors are annoying. Errors that cost you money and time must be fixed. 😜

Some web searches eventually led me to Arch Linux’s excellent Wayland wiki page. None of the adjustments or environment variables there had any effect on my cursors.

I eventually landed on a page that suggested setting XCURSOR_SIZE. I don’t remember ever setting that, but it was being set by something:

$ echo $XCURSOR_SIZE
24

One of the suggestions was to decrease it, so I decided to give 20 a try. That was too big, but 16 was perfect and it matched all of my other applications:

$ export XCURSOR_SIZE=20
# /opt/tastytrade/bin/tastytrade

That works fine when I start my application via the terminal, but how do I set it for the application when I start it from ulauncher in sway? 🤔

Desktop file #

The Tastytade RPM comes with a .desktop file for launching the application. I copied that over to my local applications directory:

cp /opt/tastytrade/lib/tastytrade-tastytrade.desktop \
 ~/.local/share/applications/

Then I opened the copied ~/.local/share/applications/tastytrade-tastytrade.desktop file in a text editor:

[Desktop Entry]
Name=tastytrade
Comment=tastytrade
Exec=/opt/tastytrade/bin/tastytrade
Icon=/opt/tastytrade/lib/tastytrade.png
Terminal=false
Type=Application
Categories=tastyworks
MimeType=

I changed the Exec line to be:

Exec=env XCURSOR_SIZE=16 /opt/tastytrade/bin/tastytrade

I launched the application again after making that change, but the cursors were still huge! There has to be another way. 🤔

systemd does everything 😆 #

After more searching and digging, I discovered that systemd has a capability to set environment variables for user sessions:

Configuration files in the environment.d/ directories contain lists of environment variable assignments passed to services started by the systemd user instance. systemd-environment-d-generator(8) parses them and updates the environment exported by the systemd user instance. See below for an discussion of which processes inherit those variables.

It is recommended to use numerical prefixes for file names to simplify ordering.

For backwards compatibility, a symlink to /etc/environment is installed, so this file is also parsed.

Let’s give that a try:

$ mkdir -p ~/.config/environment.d/
$ vim ~/.config/environment.d/wayland.conf

In the file, I added one line with a comment (because you will soon forget why you added it 😄):

# Fix big cursors in Java apps in Wayland
XCURSOR_SIZE=16

After a reboot, I launched my Java application and boom – the cursors were perfect! 🎉

I went back and cleaned up some other hacks I had applied and added them to that wayland.conf file:

# This was important at some point but I'm afraid to remove it.
# Note to self: make detailed comments when adding lines here.
SDL_VIDEODRIVER=wayland
QT_QPA_PLATFORM=wayland

# Reduce window decorations for VLC
QT_WAYLAND_DISABLE_WINDOWDECORATION="1"

# Fix weird window handling when Java apps do certain pop-ups
_JAVA_AWT_WM_NONREPARENTING=1

# Ensure Firefox is using Wayland code (not needed any more)
MOZ_ENABLE_WAYLAND=1

# Disable HiDPI
GDK_SCALE=1

# Fix big cursors in Java apps in Wayland
XCURSOR_SIZE=16

I’m told there are some caveats with this solution, especially if your Wayland desktop doesn’t use systemd to start. This is working for me with GDM launching Sway on Fedora 40.

cloud-init and dhcpcd

major@mhtx.net (Major Hayden) — Thu, 18 Apr 2024 00:00:00 +0000

We’re all familiar with the trusty old dhclient on our Linux systems, but it went end-of-life in 2022:

NOTE: This software is now End-Of-Life. 4.4.3 is the final release
planned. We will continue to keep the public issue tracker and user
mailing list open.

You should read this file carefully before trying to install or use
the ISC DHCP Distribution.

Most Linux distributions use dhclient along with cloud-init for the initial dhcp request during the first part of cloud-init’s work. I set off to switch Fedora’s cloud-init package to dhcpcd instead.

What’s new with dhcpcd? #

There are some nice things about dhcpcd that you can find in the GitHub repository:

Very small footprint with almost no dependencies on Fedora
It can do DHCP and DHCPv6
It can also be a ZeroConf client

The project had its last release back in December 2023 and had commits as recently as this week.

But I use NetworkManager #

That’s great! A switch from dhclient to dhcpcd for cloud-init won’t affect you.

When cloud-init starts, it does an initial dhcp request to get just enough networking to reach the cloud’s metadata service. This service provides all kinds of information for cloud-init, including network setup instructions and initial scripts to run.

NetworkManager doesn’t start taking action until cloud-init has written the network configuration to the system.

But I use systemd-networkd #

Same as with NetworkManager, this change applies to the very early boot and you won’t notice a different when deploying new cloud systems.

How can I get it right now? #

If you’re using a recent build of Fedora rawhide (the unstable release under development), you likely have it right now on your cloud instance. Just run journalctl --boot, search for dhcpcd, and you should see these lines:

cloud-init[725]: Cloud-init v. 24.1.4 running 'init-local' at Wed, 17 Apr 2024 14:39:36 +0000. Up 6.13 seconds.
dhcpcd[727]: dhcpcd-10.0.6 starting
kernel: 8021q: 802.1Q VLAN Support v1.8
dhcpcd[730]: DUID 00:01:00:01:2d:b2:9b:a9:06:eb:18:e7:22:dd
dhcpcd[730]: eth0: IAID 18:e7:22:dd
dhcpcd[730]: eth0: soliciting a DHCP lease
dhcpcd[730]: eth0: offered 172.31.26.195 from 172.31.16.1
dhcpcd[730]: eth0: leased 172.31.26.195 for 3600 seconds
avahi-daemon[706]: Joining mDNS multicast group on interface eth0.IPv4 with address 172.31.26.195.
avahi-daemon[706]: New relevant interface eth0.IPv4 for mDNS.
avahi-daemon[706]: Registering new address record for 172.31.26.195 on eth0.IPv4.
dhcpcd[730]: eth0: adding route to 172.31.16.0/20
dhcpcd[730]: eth0: adding default route via 172.31.16.1
dhcpcd[730]: control command: /usr/sbin/dhcpcd --dumplease --ipv4only eth0

There’s also an update pending for Fedora 40, but it’s currently held up by the beta freeze. That should appear as an update as soon as Fedora 40 is released.

Keep in mind that if you have a system deployed already, cloud-init won’t need to run again. Updating to Fedora 40 will update your cloud-init and pull in dhcpcd, but it won’t need to run again since your configuration is already set.

Texas Linux Fest 2024 recap 🤠

major@mhtx.net (Major Hayden) — Tue, 16 Apr 2024 00:00:00 +0000

The 2024 Texas Linux Festival just ended last weekend and it was a fun event as always. It’s one my favorite events to attend because it’s really casual. You have plenty of opportunities to see old friends, meet new people, and learn a few things along the way.

I was fortunate enough to have two talks accepted for this year’s event. One was focused on containers while the other was a (very belated) addition to my impostor syndrome talk from 2015.

This was also my first time building slides with reveal-md, a “batteries included” package for making reveal.js slides. Nothing broke too badly and that was a relief.

Containers talk #

I’ve wanted to share more of what I’ve done with CoreOS in low-budget container deployments and this seemed like a good time to share it with the world out loud. My talk, Automated container updates with GitHub and CoreOS, walked the audience through how to deploy containers on CoreOS, keep them updated, and update the container image source.

My goal was to keep it as low on budget as possible. Much of it was centered around a stack of caddy, librespeed, and docker-compose. All of it was kept up to date with watchtower.

My custom Caddy container needed support for Porkbun’s DNS API and I used GitHub Actions to build that container and serve it to the internet using GitHub’s package hosting. This also gave me the opportunity to share how awesome Porkbun is for registering domains, including their customized pig artwork for every TLD imaginable. 🐷

We had a great discussion afterwards about how CoreOS does indeed live on as Fedora CoreOS.

Tech career talk #

This talk made me nervous because it had a lot of slides to cover, but I also wanted to leave plenty of time for questions. Five tips for a thriving technology career built upon my old impostor syndrome talk by sharing some of the things I’ve learned over the year that helped me succeed in my career.

I managed to end early with time for questions, and boy did the audience have questions! 📣 Some audience members helped me answer some questions, too!

We talked a lot about office politics, tribal knowledge, and toxic workplaces. The audience generally agreed that most businesses tried to rub copious amounts of Confluence on their tribal knowledge problem, but it never improved. 😜

The room was full with people standing in the back and I’m tremendously humbled by everyone who came. I received plenty of feedback afterwards and that’s the best gift I could ever get. 🎁

Other talks #

Anita Zhang had an excellent keynote talk on the second day about her unusual path into the world of technology. Her slides were pictures of her dog that lined up with various parts of her story. That was a great idea.

Kyle Davis offered talks on valkey and bottlerocket. There was plenty about the redis and valkey story that I didn’t know and the context was useful. It looks like you can simply drop valkey into most redis environments without much disruption.

Thomas Cameron talked about running OKD on Fedora CoreOS in his home lab. There were quite a few steps, but he did a great job of connecting the dots between what needed to be done and why.

Around the exhibit hall #

I helped staff the Fedora/CoreOS booth and we had plenty of questions. Most questions were around the M1 Macbook running Asahi Linux that was on the table. 😉

There were still quite a few misconceptions around the CentOS Stream changes, as well as how AlmaLinux and Rocky Linux fit into the picture. Our booth was right next to the AlmaLinux booth and I had the opportunity to meet Jonathan Wright. That was awesome!

I can’t wait for next year’s event.

Roll your own static blog analytics

major@mhtx.net (Major Hayden) — Thu, 04 Apr 2024 00:00:00 +0000

Static blogs come with tons of advantages. They’re cheap to serve. You store all your changes in git. People with spotty internet connections can clone your blog and run it locally.

However, one of the challenges that I’ve run into over the years is around analytics.

I could quickly add Google Analytics to the site and call it a day, but is that a good idea? Many browsers have ad blocking these days and the analytics wouldn’t even run. For those that don’t have an ad blocker, do I want to send more data about them to Google? 🙃

How about running my own self-hosted analytics platform? That’s pretty easy with containers, but most ad blockers know about those, too.

This post talks about how to host a static blog in a container behind a Caddy web server. We will use goaccess to analyze the log files on the server itself to avoid dragging in an analytics platform.

Why do you need analytics? #

Yes, yes, I know this comes from the guy who wrote a post about writing for yourself, but sometimes I like to know which posts are popular with other people. I also like to know if something’s misconfigured and visitors are seeing 404 errors for pages which should be working.

It can also be handy to know when someone else is writing about you, especially when those things are incorrect. 😉

So my goals here are these:

Get some basic data on what’s resonating with people and what isn’t
Find configuration errors that are leading visitors to error pages
Learn more about who is linking to the site
Do all this without impacting user privacy through heavy javascript trackers

What are the ingredients? #

There are three main pieces:

Caddy, a small web server that runs really well in containers
This blog, which is written with Hugo and stored in GitHub
Goaccess, a log analyzer with a capability to do live updates via websockets

Caddy will write logs to a location that goaccess can read. In turn, goaccess will write log analysis to an HTML file that caddy can serve. The HTML file served by caddy will open a websocket to goaccess for live analytics.

A static blog in a container? #

We can pack a static blog into a very thin container with an extremely lightweight web server. After all, caddy can handle automatic TLS certificate installation, logging, and caching. That just means we need the most basic webserver in the container itself.

I was considering a second caddy container with the blog content in it until I stumbled upon a great post by Florin Lipan about The smallest Docker image to serve static websites. He went down a rabbit hole to make the smallest possible web server container with busybox.

His first stop led to a 1.25MB container, and that’s tiny enough for me.¹ 🤏

I built a container workflow in GitHub Actions that builds a container, puts the blog in it, and stores that container as a package in the GitHub repository. It all starts with a brief Dockerfile:

FROM docker.io/library/busybox:1.36.1
RUN adduser -D static
USER static
WORKDIR /home/static
COPY ./public/ /home/static
CMD ["busybox", "httpd", "-f", "-p", "3000"]

We start with busybox, add a user, put the website content into the user’s home directory, and start busybox’s httpd server. The container starts up and serves the static content on port 3000.

Caddy logs #

Caddy writes its logs in a JSON format and goaccess already knows how to parse caddy logs. Our first step is to get caddy writing some logs. In my case, I have a directory called caddy/logs/ in my home directory where those logs are written.

I’ll mount the log storage into the caddy container and mount one extra directory to hold the HTML file that goaccess will write. Here’s my docker-compose.yaml excerpt:

 caddy:
 image: ghcr.io/major/caddy:main
 container_name: caddy
 ports:
 - "80:80/tcp"
 - "443:443/tcp"
 - "443:443/udp"
 restart: unless-stopped
 volumes:
 - ./caddy/Caddyfile:/etc/caddy/Caddyfile:Z
 - caddy_data:/data
 - caddy_config:/config
 # Caddy writes logs here 👇
 - ./caddy/logs:/logs:z
 # This is for goaccess to write its HTML file 👇
 - ./storage/goaccess_major_io:/var/www/goaccess_major_io:z

Now we need to update the Caddyfile to tell caddy where to place the logs and add a reverse_proxy configuration for our new container that serves the blog:

major.io {
 # We will set up this container in a moment 👇
 reverse_proxy major_io:3000 {
 lb_try_duration 30s
 }

 # Tell Caddy to write logs to `/logs` which
 # is `storage/logs` on the host:
 log {
 output file /logs/major.io-access.log {
 roll_size 1024mb
 roll_keep 20
 roll_keep_for 720h
 }
 }
}

Great! We now have the configuration in place for caddy to write the logs and the caddy container can mount the log and analytics storage.

Enabling analytics #

We’re heading back to the docker-compose.yml file once more, this time to set up a goaccess container:

 goaccess_major_io:
 image: docker.io/allinurl/goaccess:latest
 container_name: goaccess_major_io
 restart: always
 volumes:
 # Mount caddy's log files 👇
 - "./caddy/logs:/var/log/caddy:z"
 # Mount the directory where goaccess writes the analytics HTML 👇
 - "./storage/goaccess_major_io:/var/www/goaccess:rw"
 command: "/var/log/caddy/major.io-access.log --log-format=CADDY -o /var/www/goaccess/index.html --real-time-html --ws-url=wss://stats.major.io:443/ws --port=7890 --anonymize-ip --ignore-crawlers --real-os"

This gets us a goaccess container to parse the logs from caddy. We need to update the caddy configuration so that we can reach the goaccess websocket for live updates:

stats.major.io {
 root * /var/www/goaccess_major_io
 file_server
 reverse_proxy /ws goaccess_major_io:7890
}

At this point, we have caddy writing logs in the right place, goaccess can read them, and the analytics output is written to a place where caddy can serve it. We’ve also exposed the websocket from goaccess for live updates.

Serving the blog #

We’ve reached the most important part!

We added the caddy configuration to reach the blog container earlier, but now it’s time to deploy the container itself. As a reminder, this is the container with busybox and the blog content that comes from GitHub Actions.

The docker-compose.yml configuration here is very basic:

 major_io:
 image: ghcr.io/major/major.io:main
 container_name: major_io
 restart: always

Caddy will connect to this container on port 3000 to serve the blog. (We set port 3000 in the original Dockerfile).

At this point, everything should be set to go. Make it live with:

docker-compose up -d

This should bring up the goaccess and blog containers while also restarting caddy. The website should be visible now at major.io (and that’s how you’re reading this today).

What about new posts? #

I’m glad you asked! That was something I wondered about as well. How do we get the new blog content down to the container when a new post is written? 🤔

As I’ve written in the past, I like using watchtower to keep containers updated. Watchtower offers an HTTP API interface for webhooks to initiate container updates. We can trigger that update via a simple curl request from GitHub Actions when our container pipeline runs.

My container workflow has a brief bit at the end that does this:

 - name: Update the blog container
 if: github.event_name != 'pull_request'
 run: |
 curl -s -H "Authorization: Bearer ${WATCHTOWER_TOKEN}" \
 https://watchtower.thetanerd.com/v1/update 
 env:
 WATCHTOWER_TOKEN: ${{ secrets.WATCHTOWER_TOKEN }}

You can enable this in watchtower with a few new environment variables in your docker-compose.yml:

 watchtower:
 # New environment variables 👇
 environment:
 - WATCHTOWER_HTTP_API_UPDATE=true
 - WATCHTOWER_HTTP_API_TOKEN=SUPER-SECRET-TOKEN-PASSWORD
 - WATCHTOWER_HTTP_API_PERIODIC_POLLS=true

WATCHTOWER_HTTP_API_UPDATE enables the updating via API and WATCHTOWER_HTTP_API_TOKEN sets the token required when making the API request. If you set WATCHTOWER_HTTP_API_PERIODIC_POLLS to true, watchtower will still periodically look for updates to containers even if an API request never appeared. By default, watchtower will stop doing periodic updates if you enable the API.

This is working on my site right now and you can view my public blog stats on stats.major.io. 🎉

Florin went all the way down to 154KB and I was extremely impressed. However, I’m not too worried about an extra megabyte here. 😉 ↩︎

Connect Caddy to Porkbun

major@mhtx.net (Major Hayden) — Thu, 29 Feb 2024 00:00:00 +0000

I recently told a coworker about Caddy, a small web and proxy server with a very simple configuration. It also has a handy feature where it manages your TLS certificate for you automatically.

However, one problem I had at home with my CoreOS deployment is that I don’t have inbound network access to handle the certificate verification process. Most automated certificate vendors need to reach your web server to verify that you have control over your domain.

This post talks about how to work around this problem with domains registered at Porkbun.

DNS validation #

Certificate providers usually default to verifying domains by making a request to your server and retrieving a validation code. If your systems are all behind a firewall without inbound access from the internet, you can use DNS validation instead.

The process looks something like this:

You tell the certificate provider the domain names you want on your certificate
The certificate provider gives you some DNS records to add wherever you host your DNS records
You add the DNS records
You get your certificates once the certificate provider verifies the records.

You can do this manually with something like acme.sh today, but it’s painful:

# Make the initial certificate request
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please

# Add your DNS records manually.

# Verify the DNS records and issue the certificates.
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please --renew

# Copy the keys/certificates and configure your webserver.

We don’t want to live this way.

Let’s talk about how Caddy can help.

Adding Porkbun support to Caddy #

Caddy is a minimal webserver and Porkbun support doesn’t get included by default. However, we can quickly add it via a simple container build:

FROM caddy:2.7.6-builder AS builder

RUN xcaddy build \
 --with github.com/caddy-dns/porkbun

FROM caddy:2.7.6

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

This is a two stage container build where we compile the Porkbun support and then use that new caddy binary in the final container.

We’re not done yet!

Automated Caddy builds with updates #

I created a GitHub repository that builds the Caddy container for me and keeps it updated. There’s a workflow to publish a container to GitHub’s container repository and I can pull containers from there on my various CoreOS machines.

In addition, I use Renovate to watch for Caddy updates. New updates come through a regular pull request and I can apply them whenever I want.

Example pull request from Renovate

Connecting to Porkbun #

We start here by getting an API key to manage the domain at Porkbun.

Log into your Porkbun dashboard.
Click Details to the right of the domain you want to manage.
Look for API Access in the leftmost column and turn it on.
At the top right of the dashboard, click Account and then API Access.
Add a title for your new API key, such as Caddy, and click Create API Key.
Save the API key and secrey key that are displayed.

Open up your Caddy configuration file (the Caddyfile) and add some configuration:

{
 email me@example.com

 # Uncomment this next line if you want to get
 # some test certificates first.
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

 acme_dns porkbun {
 api_key pk1_******
 api_secret_key sk1_******
 }
}

example.com {
 handle {
 respond "Hello world!"
 }
}

Save the Caddyfile and restart your Caddy server or container. Caddy will immediately begin requesting your TLS certificates and managing your DNS records for those certificates. This normally finishes in less than 30 seconds or so during the first run.

If you don’t see the HTTPS endpoint working within a minute or two, be sure to check the Caddy logs. You might have a typo in a Porkbun API key or the domain you’re trying to modify doesn’t have the API Access switch enabled.

Remember that Porkbun requires you to enable API access for each domain. API access is disabled at Porkbun by default.

That’s it! 🎉

Renewals #

Caddy will keep watch over the certificates and begin the renewal process as the expiration approaches. It has a very careful retry mechanism that ensures your certificates are updated without tripping any rate limits at the certificate provider.

Linux on the AMD ThinkPad Z13 G2

major@mhtx.net (Major Hayden) — Sun, 14 Jan 2024 00:00:00 +0000

AMD’s new Zen 4 processors started rolling out in 2022 and I’ve been watching for the mobile CPUs to reach laptops. I like where AMD is going with these chips and how they provide lots of CPU power without eating up the battery.

I recently ordered a ThinkPad Z13 Gen 2 with an AMD Ryzen 7. As you might expect, I loaded it up with Fedora Linux and set out to ensure that everything works.

This post includes all of the configurations and changes I added along the way.

Power management #

I removed the power profiles daemon that comes with Fedora by default. and replaced it with tlp. This is a great package for ThinkPad laptops as it takes care of most of the power management configuration for you with sane defaults. It also offers an easy to read configuration file where you can make adjustments.

The defaults seem to be working well so far, but my only complaint is that the power management for amdgpu seems to be really aggressive. Graphics performance on battery power is okay, but I’m told this improves in kernel 6.7. I’m on 6.6.11 in Fedora 39 right now.

I’ll wait to see if this new kernel makes any improvements.

Touchpad #

The ELAN touchpad in the Z13 is a bit different. It’s a haptic touchpad. It doesn’t push down with a click like the other thinkpads. It provides haptic feedback, much like a mobile phone does when you tap on the screen. (I usually turn this off on my phone, but it feels good on the laptop.)

The touchpad works right out of the box without any additional configuration. I made a basic Sway configuration stanza to get it configured with my preferences:

# ThinkPad Z13 Gen 2 AMD Touchpad
input "11311:40:SNSL0028:00_2C2F:0028_Touchpad" {
 drag disabled
 tap enabled
 dwt enabled
 natural_scroll disabled
}

The configuration above enables tap to click and dragging with taps. I like the old school scrolling style and I’ve disabled the natural scroll.

You can always get a list of your input devices in Sway with swaymsg:

swaymsg -t get_inputs

Display #

The display worked right out of the box but the UI elements were scaled up far too large for me. I typically value screen real estate over all other aspects, but my usual default of scaling to 1.0 made the UI far too small.

I set my output scaling to 1.2:

# Disable HiDPI
output * scale 1.2

I also enabled the RPM Fusion repos to get the freeworld AMD Mesa drivers:

sudo dnf install \
 https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
 https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
sudo dnf swap mesa-va-drivers mesa-va-drivers-freeworld
sudo dnf swap mesa-vdpau-drivers mesa-vdpau-drivers-freeworld

Audio #

Sound worked right out of the box, but I found that the loudness preset from easyeffects made the speakers sound a little bit better:

sudo dnf install easyeffects

Everything else #

Everything else just worked!

I’m really pleased with the performance and the battery life so far. My only complaint is that the OLED screen can be a battery hog at times.

For more details, check out the Arch Linux wiki page for the Z13. They documented lots of the function keys if you want to create keyboard shortcuts and they link to some downloadable monitor profiles.

Dark mode in Sway

major@mhtx.net (Major Hayden) — Tue, 09 Jan 2024 00:00:00 +0000

Ah, dark mode! I savor my dark terminals, window decorations, and desktop wallpapers. It’s so much easier on my eyes on those long work days. 😎

However, I think the author Mary Oliver said it best:

Someone I loved once gave me a box full of darkness. It took me years to understand that this too, was a gift.

In most window managers, such as GNOME or KDE, switching to dark mode involves a simple trip to the settings panels and clicking different themes. Sway doesn’t offer us those types of comforts, but we can get dark mode there, too!

GTK applications #

If you happen to have GNOME on your system alongside sway, go into Settings, then Appearance and select Dark. You can also get dark mode by applying a setting in ~/.config/gtk-3.0/settings.ini:

[Settings]
gtk-application-prefer-dark-theme=1

Restart whichever application you were using and it should pick up the new configuration.

Firefox, for example, ships with an automatic appearance setting that follows the OS. That should be reflected immediately upon restart. If not, go into Firefox’s settings, and look for dark mode under the Language and Appearance section of the general settings.

QT applications #

Most of my applications are GTK-based, but I have one or two which use QT. Again, just like the GTK example, if you have KDE installed along side Sway, you can configure dark mode there easily. Just open the system settings and look for Breeze Dark in the Plasma Style section.

You don’t have KDE? Don’t worry! There are a couple of commands which should work:

# This should work for all QT/KDE apps
# if you have the Breeze Dark theme installed.
lookandfeeltool -platform offscreen \
 --apply "org.kde.breezedark.desktop"

# You can set the theme for GTK apps here as well
# if you run into problems.
dbus-send --session --dest=org.kde.GtkConfig \
 --type=method_call /GtkConfig org.kde.GtkConfig.setGtkTheme \
 "string:Breeze-dark-gtk"

Alternate dark mode based on time #

Many window managers offer a method for adjusting dark and light modes based on the time of day. For example, some people love brighter interfaces during the day and darker ones at night. There’s a great tool called darkman that makes this easier. 🤓

The darkman service runs in the background and runs various commands to change dark mode settings for all kinds of window managers. It also speaks to dbus directly to set the configurations if needed.

It also has a directory full of user contributed scripts to change dark and light modes for various environments. You might be able to pull some commands from these files to test which configurations might work best on your system.

On diversity

major@mhtx.net (Major Hayden) — Sat, 16 Dec 2023 00:00:00 +0000

️ 👋 This post represents my own views on the topic of diversity and it doesn’t represent the views of my employer or any professional group I belong to.

I’ve written a post on diversity and deleted it several times. It remains a sensitive topic for different people for different reasons. My gut feeling is that no matter how you frame a post on diversity, some group of people will be upset about it¹.

There was a great speaker who came and spoke to us at my last job and she made an excellent point that I remember today:

Your experiences are yours. Nobody can take them away from you.

Nobody can say that your experiences do not matter.

Nobody can tell you that you didn’t experience what you experienced.

Sharing these experiences with others allows us to grow and understand more about the world around us.

That speech entirely changed my way of thinking about interactions with other people at work and at home. There are two main benefits here:

It’s incredibly freeing for someone who has experienced something to be able to share it with others and not be told that their experience was wrong or misguided.
It’s also freeing for the listener to take in someone else’s experience and be able to ask clarifying questions so they get a better understanding of how something felt for someone else.

With that in mind, here’s we go with the rest of the post. I’m not deleting it this time.

I promise. 😉

My first experience #

I’ve written in the past about my unexpected leap to lead an information security architecture team in a previous role. Being a director was a new world unto itself, but then I found that my team wasn’t performing well. To make matters worse, our success was critical to the ongoing work of the security department as a whole.

We had three members of the team that all brought something unique to the team’s perspective. All three were men of different races, but each had a different approach to security based on their backgrounds and experiences. One left the team due to some interpersonal issues that eventually boiled over.

I was suddenly down to two people and our team needed to hire two as soon as possible.

Recruiting teams started putting feelers out into the market to find talented people and I was poking several friends for referrals. A colleague in another department reached out and really wanted to join the team. I knew about her experience from several previous interactions and she was highly recommended from her peers. She joined the team and hit the ground running.

As applicants began trickling in through the recruiting team, I started with screening calls for each. We really needed someone with skills in a few key areas:

Great communicator with empathy
Knowledge of secure development and operations practices
Someone who could be trusted to operate independently and work on team projects

Most of the applicants I screened were male and that wasn’t a surprise at the time. We brought five through the screening into interviews and it was down to four males and one female. Three of them turned out to be great and they all had deep knowledge of security architecture. After another round of interviews, we began to realize that the female matched the other applicants, but her communication skills were stronger, especially under pressure.

Needless to say, we sent her the offer and she accepted! We were thrilled! Our team was full!

Getting underway #

We began chipping away at the mountain of projects set aside for our team and started making progress. Our new team member was struggling to move from the rigidity of her previous employer to our new way of working, but she adjusted well over time.

I sat down in one of our weekly leadership meetings some time later. These meetings usually involved a round-the-horn of what’s working well for each team, the threats on the board for the next few months, and our plans.

We usually had an attendee from HR in the meetings for various reasons and she asked me how our new team member was doing. I said:

Oh, she’s doing a good job. Her last company was pretty rigid and things are different here, but she’s figuring it out. She knows her stuff and she’s a team player. We’re working through some small things here and there.

Then the HR representative said:

Well, I have to commend you for building out a such a diverse team. It’s much more so than the other teams. That’s really great work and I want to make sure you’re recognized for it.

I smiled and thanked her (because that’s my usual response), but then I almost felt sick.

Different view #

I left that meeting and went back to my desk to think.

Had I assembled a diverse team intentionally? No, I didn’t. I looked for people who had the qualities we desperately needed, gave them guardrails, and got out of the way. That’s what you do with smart people, right?

Then I wondered, “Is my team really diverse?”

There were two men and two women.
Two were on the younger end of a generation and two were on the older end.
One was of mixed Asian descent, one was Hispanic, and two were what most people would likely refer to as “white.”

So maybe my team is diverse.

Then I realized that most of the people on the team had the same certifications, all had at least an undergraduate degree, and all were married. All of them were in heterosexual relationships and all dressed in a way that aligned with their gender.

Does that mean they’re not diverse?

Is my team more or less diverse than other teams?

Does any of this even matter?

Did I do a good or a bad thing?

Diversity challenges #

This brings me to the two problems I struggle with most around diversity, especially when people talk about increasing or improving diversity on their team or within their company:

Quantifying diversity is highly subjective and in the eye of the beholder.
Challenges arise when you apply diversity requirements to real world situations.

I’ll break down both of these now.

In the eye of the beholder #

You can choose how you want to measure diversity on all kinds of factors. Depending on the factors, a team can look more or less diverse. Also, your experiences often define how you judge the diversity of people and teams.

One could argue that a team made up entirely of white males is likely not very diverse. The majority of people would likely agree with that statement.

However, what if those males vary in their sexual orientations, educational backgrounds, and socioeconomic status. Is that diverse?

If you have a team of people made up of various genders with various sexual orientations from all contents on the planet, but they all went to Ive League schools and they’re all wealthy – is that diverse?

Are any of these examples diverse enough? Does the answer to that question even matter?

In my experience, assembling a team of people with different backgrounds and approaches to problems is incredibly valuable. That type of diversity led to some incredible innovation in the past.

However, these diverse backgrounds and approaches don’t always line up with differences in gender identity, socioeconomic status, sexual orientation, or other factors. This is why I find it really challenging to quantify the level of diversity within a company or in individual teams.

Rubber meets the road #

There’s a common phrase in English: “when the rubber meets the road.”

In a literal sense, it’s referring to when car tires move on pavement during a race. What it really means is, when it comes time to do something for real and the stakes are high, what happens?

Here’s another example. Let’s say you lead an engineering team that is all males and your company says that diversity must be a priority in hiring decisions.

So you take your job requisition and send it through the recruiting team. They work hard to remove any gender-specific language or anything else that might turn an applicant away. You put the job on the internet, talk to your friends about people they know, and then wait for the responses.

Let’s assume you get ten male applicants.

Do you proceed with screening and interviewing them while you try harder to drum up more female applicants? If a female applicant never appears, do you pause the hiring process while you try to find one? What if your existing applicants find other roles in the meantime and suddenly your applicant pipeline is empty?

Some might say “Yes, of course you wait until you can find a female applicant!” In that case, your team is still short-staffed and likely not performing as well as it could. Would that be good for your customers? How about your shareholders?

Others might say “No, go ahead and complete the hiring process but you should search harder for women for future roles.” In this case, you’ll have a fully staffed team and hopefully be delivering more value quickly. However, you haven’t improved the diversity on your team and that could come back to be a problem if you’re asked about it later.

Go backwards a bit with the same example and assume you get a split of ten applicants: half male and half female. That’s awesome because now you have a diverse talent pool, right?

Here’s where it gets challenging.

If you interview them all and make an offer to the female applicant because she has the skills and qualifications needed, you now have a more diverse team (on one measure) and you’re fully staffed! Great!

If you interview them all and it turns out one of the men has the best skills and qualifications, what do you do? Your company made diversity a priority, but you’re also trying to assemble a strong team.

Do you take a less qualified applicant that improves the team’s diversity?

Or, do you take a more qualified applicant that leaves the team’s diversity unchanged?

This is where diversity breaks down: when you have to really sit down and compare outcomes, there’s not a right answer.

Another viewpoint #

My wife constantly points out things to me that I completely missed and we’ve talked about this topic many times. She has asked me the same thing in the past:

Why do people in your field care so much about getting women into technology? I hate technology. Maybe other women hate technology, too. If I knew I was hired someplace because they wanted a woman for the role and they weren’t looking at how well someone could do the job, I’d be pretty upset.

She’s a medical professional and she’s happy to remind me about this:

I went to PA (physician assistant) school and most people there were women. All the nurses at my office are women. All the front office staff are women. We’re not out there trying to get male nurses or male front office staff in here all the time. We just find people who do their job well and hire them.

Our conversations really make me stop and think.

My goals #

I’d also like to see more people from underrepresented communities across the globe break into the world of technology and really change things. This means empowering a wider array of people with varying gender, education, nationality, wealth, and opportunities to join a field of work which they thought might be inaccessible to them.

This is why I try to volunteer as much as possible to inspire young people of all backgrounds to set goals for themselves and look at the world as if nothing is out of reach.

It’s one of the reasons I write this blog and put everything out there for free. Democratizing access to learning (and my mediocre blog posts) is key to leveling the playing field.

These are some of those pieces of work that are never finished.

However, I really worry that quantifying diversity or forcing one’s definition of diversity onto someone else could lead us to a bad place where no result is satisfactory. It’s much more subjective than some would like to admit and that becomes a problem when you directly apply it to specific situations.

In the meantime, I’ll keep writing these posts, mentoring others, and lifting people up to do things they never imagined they could do. ️♥️

Then again, we live in a world where someone can say “Puppies are cute” and the first reply would be “Why do you hate cats so much?” 😄 ↩︎

Horror book reviews from October 2023

major@mhtx.net (Major Hayden) — Sun, 19 Nov 2023 00:00:00 +0000

Reading allows me to travel to other places and times while also reducing my stress and helping me to think more creatively. Sometimes this leads me to wild fictional stories or takes me on a learning journey into history.

(I track my reading lists on Goodreads if you want to see what I’m reading.)

In this post, I’ll list the spooky books I read this October and hopefully you’ll find at least one them interesting!

The Cabin at the End of the World #

My first book of the month was Paul Tremblay’s The Cabin at the End of the World. It’s centered around a family with a small child that goes on a relaxing vacation in a lakefront cabin.

All is well until a friendly stranger named Leonard befriends the child, Wen, and his three fellow travelers appear. The travelers hold the family hostage and tell them that they have the key to save the world, but it’s not as easy as it seems. There seems to be an unseen force that has some sort of control over the travelers. 🤔

My thoughts #

I thought I had this story figured out several times only to be proven wrong just as many times. There are so many themes in this book that tug at your emotions, including family, racism, homophobia, and socioeconomic differences. This book packs plenty of suspense, but the most frightening parts aren’t supernatural or ghostly. The scariest parts feel centered around the essence of human nature.

Although this felt like a quick read, it was intense in many places. I definitely enjoyed it and I was looking forward to seeing the movie adaptation, Knock at the Cabin. The movie was done by M. Night Shyamalan (of The Sixth Sense fame) and I read that he changed the entire ending of the movie. 😞

The Ruins #

I moved onto something completely different next with Scott Smith’s The Ruins. The story takes place in Mexico with several people on a beach vacation. One of the tourists notes that his brother went to check out an archeological dig further into the countryside but never returned. The group eventually decides to go search for the missing brother as some sort of vacation adventure.

The adventure turns ugly as they make their way to the country’s interior. 😬

My thoughts #

You’ll have a tough time finding the antagonist in this story until it’s too late, but that’s part of the fun. Every character in the group brings personality quirks and old baggage with them that impairs their judgement in different ways. This works well for some but not for others.

This book had several scary moments, but most of the horror came again from how humans interact with one another. As soon as someone (or something) else figures out how to exploit those against them, bad things happen.

This book was difficult to read because much of it was quite gruesome and brutal. It’s definitely a book for adults only and you should be prepared to work your way through it.

The Troop #

Everything is fine with a scout trip to an island in Canada in Nick Cutter’s The Troop. Well, it’s fine until a mysterious and incredibly hungry man suddenly shows up on the island. He looks a lot like he’s dead already and the the leader of the scout troop, a medical doctor, is completely mystified by the man’s condition.

It goes downhill from there in a story told mostly through diary entries, newspaper clippings, and court testimony.

My thoughts #

Of all the books I read in October, this one scared me the most because it felt like it was entirely possible. There wasn’t any part of the book that I looked at and said: “Oh, that could never happen.” That’s what makes this one so good.

It’s suspenseful enough to keep you turning the pages but it’s also plausible enough that you might find yourself wanting to wash your hands a little longer the next time you eat a meal. It feels a bit like Lord of the Flies mixed with a pandemic novel like Station Eleven or The Stand.

This one is also very gruesome in parts with some very difficult scenes to read.

This one was my favorite of the group by far.

Devolution: A Firsthand Account of the Rainier Sasquatch Massacre #

Sasquatch is back in Max Brooks’ Devolution! Told through the journals of a woman living in a remote town in Washington, this book covers a modern time where Mount Ranier erupted and caused lots of species to get on the move from their habitats around the mountain. Some of those creatures are the ones you’d expect, but some are ones you won’t expect.

The small community is an experiment in green, off the grid living, and they are quickly tested by just about everything mother nature can throw at them. This includes some rather tall, furry, human-like creatures.

My thoughts #

This one felt scary due to the remoteness of the village and the unprepared people involved. Also, whether you believe in Sasquatch or not, this felt a bit more plausible than I expected.

There were plenty of difficult to read scenes in here, but the gore was reduced compared to other books listed here. Much of the suspense came from how humans interact with one another especially when faced with an adversary that presents a unique set of challenges.

This book was difficult to get into (it starts slow), but stick with it. It’s a wild ride.

Tender is the Flesh #

Be prepared when you crack open Agustina Bazterrica’s Tender is the Flesh. Imagine a world where animals somehow contract a virus that they quickly spread to each other and to humans as well. Pets are banned and animals near cities are killed. The pandemic puts a significant dent in the human population.

However, with all of the animals either infected or gone, where do people get meat to eat? 🤔

My thoughts #

This was by far the most challenging book to read out of the group because I honestly felt like I was going to be sick in places. The author doesn’t set out for cheap scares or basic unsettling events – there’s something much deeper. It really makes you question how humanity operates and how quickly boundaries can shift when hunger becomes a problem.

I had to take a lot of breaks with this book. It has some suspenseful parts but this book has a slow-burn horror feel that gives you hope, crushes that hope, and then starts the cycle once more.

I strongly recommend this book for adults only but I feel terrible recommending it at the same time. 😄

What’s next? #

After all of that horror and unsettling fiction, I’m shaking things up for November. My current book is Larry McMurtry’s Lonesome Dove. So many people have recommended it to me as a great book about the American west and I’m enjoying it so far.

Moving to cloud is more than just a purchasing exercise

major@mhtx.net (Major Hayden) — Fri, 27 Oct 2023 00:00:00 +0000

Much of my work at Red Hat revolves around the RHEL experience in public clouds. I thrive on input from customers, partners, and coworkers about how they consume public clouds and why they made decisions to deploy there.

Throughout this process, I run into some wild misconceptions about public clouds and what makes them useful. One that I hear most often is:

Businesses are moving to the cloud to reduce cost and improve efficiency. It’s mainly just a purchasing exercise.

This couldn’t be further from the truth.

Cloud offers a chance to start over #

Sometimes businesses find themselves in an IT quagmire¹. No matter what they do to improve their situation, it just gets worse. Capital expenditures grow and grow, datacenter space gets more expensive, and companies spend more time focusing on IT rather than their core business.

Deploying in clouds offers that chance to break the capital expense cycle and gradually improve infrastructure. The key word here is gradual.

Businesses can choose how much they want to deploy and when without worrying about expensive servers in the datacenter waiting to be used. Some deployments are greenfield, or entirely net new applications. Some are basic migrations of applications from servers or virtual machines directly to the cloud.

Either way, businesses have the freedom to deploy as little or as much as they want on their own schedule.

Cloud offers a chance to software-define (nearly) anything #

Anyone who has worked in a large organization before knows the pain of change management. Sure, it ticks a box on that yearly compliance program, but it also ensures that everyone is aligned on the plan.

One of the greatest aspects of cloud is that you can define almost everything in software. This makes changes easier to apply, easier to roll back, and easier to track.

Tools like Terraform or Ansible allow developers and operations team to work from the same playbook. My team enjoys using Infracost to track how much a particular Terraform change might cost us under different scenarios.

Once teams set a policy of “we define our changes in git, and that’s it”, you can rely on a git history for change management. This avoids drift in production environments and it also ensures that changes made in development environments make it into staging and then into production. The days of “it worked on my system, what’s wrong with production?” slowly fade away.

Less than ideal architectural decisions can also be adjusted over time to fit the applications being deployed. Did you set up a network incorrectly? Did you choose an instance type without enough RAM?

That’s okay!

Just adjust the deployment in git, test it in staging, and push it to production.

Cloud offers managed services #

One thing I tell people constantly is that if you bend the cloud to fit your application, you will almost always pay more. You get cost and performance efficiencies if you bend your application to fit the cloud. Confused? I’ll explain.

When I talk to people doing their first cloud deployments, they deploy everything into VMs, much as they would in a local virtualized environment.

You need a database server? Make a couple of VMs and set up replication.
You need to run a batch job via cron? Deploy a VM and add it to the crontab.
You need a server to export an NFS share? Deploy a VM with lots of storage and export it to other instances.

Do you see a pattern here?

Most public clouds offer tons of services that lift the management burden from engineering teams and offload into a managed service. For example, that cron job might be able to move into a “serverless”² service, such as AWS Lambda. It’s critical to check the pricing here to ensure you’re not headed down a bad path, but you have one less VM to maintain, one less IPv4 address to pay for, and a greatly reduced risk of configuration drift. That reduction in stress and risk might be worth any additional costs.

Deployment decisions become much easier and lower stress when you consume services offered by the provider. There are those situations where deploying a whole VM is needed, but I’ve managed to avoid that for some of my team’s recent deployments.

Our last deployment uses GitHub Actions, S3, and CloudFront and costs us about $6.50 per month to run. There are no virtual machines. There’s nothing to patch.

This blog runs on a similar stack and costs me about $0.25 per month to run.

Cloud offers geographic distribution #

Nearly every public cloud, even the smallest ones, offer you the same or similar services in a wide variety of geographic regions. Disaster recovery feels more attainable when you can easily deploy to multiple regions with the same software-defined infrastructure.

Data sovereignty continues to grow in importance around the world as more countries demand that their data remains within their borders. As long as your cloud offers a region in that country, you can deploy there. There’s no challenging legal issues with finding datacenter space or getting hardware delivered. You just change your region and deploy.

Cloud regions also allow you to bring your applications much closer to the people who use them. Reduced latency delivers content faster to customers and provides a responsive experience.

Clouds offer purchasing efficiency #

Wait a minute! Didn’t I say that moving to cloud isn’t just a purchasing exercise? 🤔

Your move to cloud should not be solely based on cutting costs or making purchasing IT more efficient. Most teams find that moving to cloud is more expensive than they anticipated because they’re finally able to get access to the right amount of resources that they need. (Also, they usually go with some more expensive options up front until they figure out how to optimize for cost.)

First off, it’s much easier to budget and pay one vendor for multiple services than deal with multiple independent vendors. Instead of paying for datacenter space, then paying for servers, then paying for network equipment, then paying for people to set it up, and so on, you pay the cloud provider for all of it.

This also extends to other purchases on the cloud, such as products from certain vendors. For example, you can buy Red Hat products directly from some cloud providers and that gets added onto your cloud invoice. You can even deploy your own Cisco ASA in the cloud if you feel so inclined.

With all of these purchases going through one vendor, you can also negotiate discounts if you set a spending commitment. Discounts depend on your committed spend, of course, and the term that you agree to spend it. There’s a whole industry around financial operations in the cloud, called FinOps, and this is one of many things that factors into it.

Wrapping up #

Public clouds offer an incredible amount of opportunity to get your IT deployments into better shape with better change control and a solid software-defined workflow. They also offer the ability to “write one check” to consume infrastructure via utility billing.

However, public clouds are not ideal for every application or situation.

Do I think that every company in the world could benefit from getting some part of their IT deployments into a public cloud platform? Yes, I do.

Would every company benefit from putting most of their infrastructure into public clouds? Very unlikely.

Some applications still benefit from being on purpose-built hardware or in certain locations where a cloud might not exist today. Clouds can also be extremely expensive if you run large workloads around the clock. They can also be painful for applications with very strict or special requirements that don’t fit a cloud deployment model well.

The vendors that will succeed the most in the cloud space are the ones that look beyond purchasing efficiencies and IT acquisition concerns. Simply dragging the old world of physical servers or virtual machines into cloud won’t lead anywhere.

Those companies that help their customers benefit from the best of what public clouds have to offer in the most secure, reliable, and simple ways will be in the driver’s seat.

A quagmire is something that gets worse no matter how you try to improve it. The only way to win is to avoid it entirely. ↩︎
Boy, I still dislike that serverless term so much. 🤦‍♂️ ↩︎

How I learned to stop worrying and love the CoreOS

major@mhtx.net (Major Hayden) — Fri, 13 Oct 2023 00:00:00 +0000

It’s quite clear that I’ve been on a CoreOS blogging streak lately. I keep getting asked by people inside and outside my company about what makes CoreOS special and why I’ve switched over so many workloads to it.

The answer is pretty basic. It makes my life easier.

I’m a Dad. I’m on the PTC (Parent Teacher Club) at one of my children’s schools. I volunteer as an IT person for a non-profit. I write software. I have other time consuming hobbies, such as ham radio, reading, and becoming a longer distance runner¹.

My available time for my own IT projects is extremely limited and CoreOS plays a part in keeping that part of my life as efficient as possible.

That’s what this blog post is about!

Updates #

First and foremost, I love how CoreOS does updates. I encourage you to read the docs on this topic, but here’s a short explanation:

Updates are automatically retrieved and they’re loaded into a slot.
Your system reboots into the new update but your original OS tree remains in place.
Did the update boot? Awesome. You’re good to go.
Did something break? The system reverts back to the known good tree.

In this way, it’s a lot like your smartphone.

You have full control over when a node looks for an update and how often it checks for them. Check out the Zincati docs for tons of controls over updates and reboots.

Some of mine are timed so well that I set maintenance windows with my monitoring provider when I know an update might take place. The updates come through, monitoring shuts off, the node reboots, and monitoring comes back. The nodes almost always come back before the monitoring even alerts me.

It also removes the reminders I would set for myself to update packages and run reboots. I know that my CoreOS nodes will do this automatically, so I don’t need to think about it.

Also, updates are rarely ever impactful to my workloads since all of them are running inside containers. My containers come right back up as soon as the node finishes its reboot.

Toolbox #

To be fair, you can get toolbox running on lots of different Linux distributions outside of CoreOS, but that’s the first place I ever used it. Toolbox, also called toolbx, gives you a utility container on your CoreOS node for all kinds of adminsitrative and diagnostic capabilities.

You might need a certain package for diagnosting a hardware issue or you might want to install some helpful utilities for the command line. Do that in a toolbox container. Just run toolbox enter and if you’ve never created a toolbox container before, you’ll get a Fedora container that matches your CoreOS release.

But it gets better.

Toolbox automatically saves your container when you’re done with it so all of your installed packages stay there for next time. Also, these containers have seamless access to anything you have in your home directory, including sockets. You’re running inside a container, but it’s almost like you’re running on the host itself inside your home directory. You get the best of both worlds.

Don’t want Fedora? You have lots of distribution options through toolbox. Read the details on custom images to create your own!

Layering #

Okay, there are those situations where you really want a package on CoreOS and toolbox might not be sufficient. My muscle memory for vim is so strong and CoreOS only comes with vi.

You have a couple of options here:

Run sudo rpm-ostree install vim, reboot, and you have vim
Run sudo rpm-ostree install --apply-live vim and you have vim right now! (And it’s there after a reboot as well.)

When a new update comes down for the base OS from CoreOS, any packages you’ve added will be layered on the base image and available after a reboot. Layering is generally chosen as a last resort option for adding packages to the system but you shouldn’t run into issues if you’re installing small utilities or command line tools.

Declarative provisioning #

If you’ve provisioned Linux distributions on cloud instances in the past, you’ve likely provided metadata that cloud-init uses to provision your system. CoreOS has something that acts a lot earlier in the boot process and has more power to get things done: ignition.

There’s a handy butane file forma that you use for writing your configuration. You use the butane utility to get it into ignition format. The ignition format is highly compressed to ensure you can fit your configuration into most cloud providers’ metadata fields.

For a real example of what you can do with ignition, check out my quadlets post where I provisioned an entire Wordpress container stack using a single ignition file.

There’s lots of documentation for writing butane configuration files for common situations. It’s easy to add files, configure Wireguard, set up users, and launch containers immediately on the first boot.

Pets and cattle #

CoreOS works well for systems that I only need online for a short time. These might be situations where I need to test a few containers and throw it away. There’s no OS to mess with and no updates to worry about.

It also works well for systems that I keep online for a long time. I have a few physical systems at home that run CoreOS and they’ve been extremely stable. I also have cloud instances on Hetzner, VULTR, and Digital Ocean that have run CoreOS for months without issues.

Quadlets might make me finally stop using docker-compose

major@mhtx.net (Major Hayden) — Mon, 25 Sep 2023 00:00:00 +0000

I’ve written a lot about containers on this blog. Why do I love containers so much?

They start quickly
They make your workloads portable
They disconnect your application stack from the OS that runs underneath
You can send your application through CI as a single container image
You can isolate workloads on the network and limit their resource usage much like a VM

However, I’m still addicted to docker-compose. Can podman’s quadlets change that?

Yes, I think they can.

What’s a quadlet? #

Podman introduced support for quadlets in version 4.4 and it’s a simpler way of letting systemd manage your containers. There was an option in the past to have podman generate systemd unit files, but those were unwieldy and full of podman command line options inside a unit file. These unit files weren’t easy to edit or even parse with eyeballs.

Quadlets make this easier by giving you a simple ini-style file that you can easily read and edit. This blog post will include some quadlets later, but here’s an example one for Wordpress:

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Lots of the lines under [Container] should look familiar to most readers who have worked with containers before. However, there’s something new here.

Check out the AutoUpdate=registry line. This tells podman to keep your container updated on a regular basis with the upstream container registry. I’ve used watchtower in the past for this, but it requires a privileged container and it’s yet another external dependency.

Also, at the very end, you’ll see a WantedBy line. This is a great place to set up container dependencies. In this example, the container that runs caddy (a web server) can’t start until Wordpress is up and running.

So why not stick with docker-compose? #

There’s no denying that docker-compose is an awesome tool. You specify the desired outcome, tell it to bring up containers, and it gets containers into the state you specified. It handles volumes, networks, and complicated configuration without a lot of legwork. The YAML files are pretty easy to read, too.

However, as with watchtower, that’s another external dependency.

My container deployments are often done at instance boot time and I don’t make too many changes afterwards. I found myself using docker-compose for the initial deployment and then I didn’t really use it again.

Why not remove it entirely and use what’s built into CoreOS already?

Quaint quadlets quickly! #

Before we start, we’re going to need a few things:

An easy to read butane configuration which gets transformed into a tiny ignition configuration for CoreOS
Some quadlets
Extra system configuration
A cloud provider with CoreOS images (using VULTR for this)

I’ve packed all of these items into my quadlets-wordpress repository to make it easy. Start by looking at the config.butane file.

Let’s break it down here. First up, we add an ssh key for the default core user.

variant: fcos
version: 1.5.0
passwd:
 users:
 - name: core
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0

Next up, we enable the podman-auto-update.timer so we get container updates automatically:

storage:
 links:
 - path: /home/core/.config/systemd/user/timers.target.wants/podman-auto-update.timer
 target: /usr/lib/systemd/user/podman-auto-update.timer
 user:
 name: core
 group:
 name: core

Next is the long files section:

 files:
 # Ensure the `core` user can keep processes running after they're logged out.
 - path: /var/lib/systemd/linger/core
 mode: 0644
 
 # Allow caddy to listen on 80 and 443.
 # Allow it to ask for bigger network buffers, too.
 - path: /etc/sysctl.d/90-caddy.conf
 contents:
 inline: |
 net.ipv4.ip_unprivileged_port_start = 80
 net.core.rmem_max=2500000
 net.core.wmem_max=2500000 

 # Set up an an environment file that containers can read to configure themselves.
 - path: /home/core/.config/containers/containers-environment
 contents:
 inline: |
 MYSQL_DATABASE=wordpress
 MYSQL_USER=wordpress
 MYSQL_ROOT_PASSWORD=mariadb-needs-a-secure-password
 MYSQL_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_HOST=mariadb
 WORDPRESS_DB_USER=wordpress
 WORDPRESS_DB_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_NAME=wordpress 
 mode: 0644

 # Deploy the caddy configuration file from the repository.
 - path: /home/core/.config/caddy/Caddyfile
 contents:
 local: caddy/Caddyfile
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add some named volumes for caddy and wordpress.
 - path: /home/core/.config/containers/systemd/caddy-config.volume
 contents:
 inline: |
 [Volume] 
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/caddy-data.volume
 contents:
 inline: |
 [Volume] 
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/wordpress.volume
 contents:
 inline: |
 [Volume] 
 user:
 name: core
 group:
 name: core

 # Create a network for all the containers to use and enable the
 # DNS plugin. This allows containers to find each other using
 # the container names.
 - path: /home/core/.config/containers/systemd/wordpress.network
 contents:
 inline: |
 [Network]
 DisableDNS=false
 Internal=false 
 user:
 name: core
 group:
 name: core

 # Add the wordpress container.
 - path: /home/core/.config/containers/systemd/wordpress.container
 contents:
 local: quadlets/wordpress.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the MariaDB container.
 - path: /home/core/.config/containers/systemd/mariadb.container
 contents:
 local: quadlets/mariadb.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the caddy container.
 - path: /home/core/.config/containers/systemd/caddy.container
 contents:
 local: quadlets/caddy.container
 mode: 0644
 user:
 name: core
 group:
 name: core

The Caddyfile is also in the repository and will be deployed by the butane configuration shown above.

We can go through each quadlet in detail. First up is MariaDB. We tell systemd that the wordpress container will want to have this one started first.

[Unit]
Description=MariaDB Quadlet

[Container]
Image=docker.io/library/mariadb:11
ContainerName=mariadb
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=mariadb.volume:/var/lib/mysql
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=wordpress.service multi-user.target default.target

The wordpress quadlet is much the same as the MariaDB one, but we tell systemd that caddy will want wordpress started first.

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Finally, the caddy quadlet contains four volumes and some published ports. These ports will be published to the container host. Also, you’ll note that the wordpress volume is mounted here, too. This is because caddy can serve static files much faster than wordpress can.

[Unit]
Description=Caddy Quadlet

[Container]
Image=docker.io/library/caddy:latest
ContainerName=caddy
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=caddy-data.volume:/data
Volume=caddy-config.volume:/config
Volume=/home/core/.config/caddy/Caddyfile:/etc/caddy/Caddyfile:Z
Volume=wordpress.volume:/var/www/html
PublishPort=80:80
PublishPort=443:443
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

Launch the quadlets #

There’s a launch script that ships this configuration to VULTR and launches a CoreOS instance:

#!/bin/bash
# This command starts up a CoreOS instance on Vultr using the vultr-cli
vultr-cli instance create \
 --os 391 \
 --plan vhp-1c-1gb-amd \
 --region dfw \
 --notify true \
 --ipv6 true \
 -u "$(butane --files-dir . config.butane)" \
 -l "coreos-$(date "+%s")"

To launch an instance, get your VULTR API key first. Then install vultr-cli and butane:

$ sudo dnf -y install butane vultr-cli

After launch, check to see what your containers are doing:

[core@vultr ~]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afa2d6501593 docker.io/library/caddy:latest caddy run --confi... 54 seconds ago Up 53 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp caddy
460426f39e6c docker.io/library/mariadb:11 mariadbd 35 seconds ago Up 35 seconds mariadb
92ece6538d5a docker.io/library/wordpress:fpm php-fpm 28 seconds ago Up 29 seconds wordpress

We should be able to talk to wordpress through caddy on port 80:

[core@vultr ~]$ curl -si http://localhost/wp-admin/install.php | head -n 25
HTTP/1.1 200 OK
Cache-Control: no-cache, must-revalidate, max-age=0
Content-Type: text/html; charset=utf-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Server: Caddy
X-Powered-By: PHP/8.0.30
Date: Mon, 25 Sep 2023 21:43:40 GMT
Transfer-Encoding: chunked

<!DOCTYPE html>
<html lang="en-US" xml:lang="en-US">
<head>
	<meta name="viewport" content="width=device-width" />
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
	<meta name="robots" content="noindex,nofollow" />
	<title>WordPress &rsaquo; Installation</title>
	<link rel='stylesheet' id='dashicons-css' href='http://localhost/wp-includes/css/dashicons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='http://localhost/wp-includes/css/buttons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='http://localhost/wp-admin/css/forms.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='http://localhost/wp-admin/css/l10n.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='http://localhost/wp-admin/css/install.min.css?ver=6.3.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui language-chooser">
<p id="logo">WordPress</p>

Awesome! 🎉

Managing containers #

Containers will automatically update on a schedule and you can check the timer:

[core@vultr ~]$ systemctl status --user podman-auto-update.timer
● podman-auto-update.timer - Podman auto-update timer
 Loaded: loaded (/usr/lib/systemd/user/podman-auto-update.timer; enabled; preset: disabled)
 Active: active (waiting) since Mon 2023-09-25 21:41:31 UTC; 3min 14s ago
 Trigger: Tue 2023-09-26 00:04:46 UTC; 2h 20min left
 Triggers: ● podman-auto-update.service

Sep 25 21:41:31 vultr.guest systemd[1786]: Started podman-auto-update.timer - Podman auto-update timer.

Quadlets are just regular systemd units:

[core@vultr ~]$ systemctl list-units --user | grep -i Quadlet
 caddy.service loaded active running Caddy Quadlet
 mariadb.service loaded active running MariaDB Quadlet
 wordpress.service loaded active running Wordpress Quadlet

As an example, you can make changes to caddy’s config file and restart it easily:

[core@vultr ~]$ systemctl restart --user caddy
[core@vultr ~]$ systemctl status --user caddy
● caddy.service - Caddy Quadlet
 Loaded: loaded (/var/home/core/.config/containers/systemd/caddy.container; generated)
 Drop-In: /usr/lib/systemd/user/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Mon 2023-09-25 21:46:28 UTC; 5s ago
 Main PID: 2652 (conmon)
 Tasks: 18 (limit: 1023)
 Memory: 15.1M
 CPU: 207ms

If you need to change a quadlet’s configuration, just open up the configuration file in your favorite editor under ~/.config/containers/systemd, reload systemd, and restart the container:

$ vi ~/.config/containers/systemd/caddy.container

--- make your edits and save the quadlet configuration ---

$ systemctl daemon-reload --user
$ systemctl restart --user caddy

Enjoy!

Mounting the AWS Elastic File Store on Fedora

major@mhtx.net (Major Hayden) — Wed, 13 Sep 2023 00:00:00 +0000

I package a few things here and there in Fedora and one of my latest packages is efs-utils. AWS offers a mount helper for their Elastic File System (EFS) product on GitHub.

In this post, I’ll explain how to:

Launch a Fedora instance on AWS EC2
Install efs-utils and launch the watchdog service
Create an EFS volume in the AWS console
Mount the EFS volume inside the Fedora instance

Always check the pricing for any cloud service before you use it! EFS pricing is based on how much you store and how often you access it. Backups are also enabled by default and they add to the monthly charges.

Let’s go! 🚀

Wait, what is EFS? #

When you launch a cloud instance (virtual machine) on most clouds, you have different storage options available to you:

Block storage: You can add partitions to this storage, create filesystems, or even use LVM. It looks like someone plugged in a disk to your instance. You get full control over every single storage block on the volume. An example of this is Elastic Block Storage (EBS) on AWS.
Object storage: Although you can’t mount object storage (typically) within your instance, you can read/write objects to this storage via an API. You can upload nearly any type of file you can imagine as an object and then download it later. Objects can also have little bits of metadata attached to them and some of the metadata include prefixes which give a folder-like experience. AWS S3 is a good example of this.
Shared filesystems: This storage shows up in the instance exactly as it sounds: you get a shared filesystem. If you’re familiar with NFS or Samba (SMB), then you’ve used shared filesystems already. They give you much better performance than object storage but offer less freedom than block storage. They’re also great for sharing the same data between multiple instances.

Using EFS is almost like having someone else host a network accessible storage (NAS) device within your cloud deployment.

Launching Fedora #

Every image in AWS has an AMI ID attached to it and you need to know the ID for the image you want in your region. You can find these quickly for Fedora by visiting the Fedora Cloud download page. Look for AWS in the list, click the button on that row, and you’ll see a list of Fedora AMI IDs. Click the rocket (🚀) for your preferred region and you’re linked directly to launch that instance in AWS!

I’m clicking the launch link for us-east-2 (Ohio)¹. To finish quickly, I’m choosing all of the default options and using a spot instance (look inside Advanced details at the bottom of the page).

Wait for the instance to finish intializing and access it via ssh:

$ ssh fedora@EXTERNAL_IP
[fedora@ip-172-31-2-38 ~]$ cat /etc/fedora-release 
Fedora release 38 (Thirty Eight)

Success! 🎉

Prepare your security group #

Before leaving the EC2 console, you need to make a note of the security group that you used for this instance. That’s because EFS uses security groups to guard access to volumes. Follow these steps to find it:

Click Instances on the left side of the EC2 console.
Click on the row showing the instance we just created.
In the bottom half of the screen, click the Security tab.
Look for Security groups in the security details and copy the security group ID for later.

It should be in the format sg-[a-f0-9]*.

If you click the security group name (after saving it), you’ll see the inbound rules associated with that security group. By default, items in the same security group can’t talk to each other. We need to allow that so our EFS mount will work later.

Click Edit inbound rules and do the following:

Click Add rule.
Choose All traffic in the Type column. (You can narrow this down further later.)
In the source box, look for the security group you just created along with your EC2 instance. If you took the default during the EC2 launch process, it might be named launch-wizard-[0-9]+.
Click Save rules.

Installing efs-utils #

Let’s start by getting the efs-utils package onto our new Fedora system:

$ sudo dnf -qy install efs-utils
Installed:
 efs-utils-1.35.0-2.fc38.noarch

The package includes some configuration, a watchdog, and a mount helper:

$ rpm -ql efs-utils
/etc/amazon
/etc/amazon/efs
/etc/amazon/efs/efs-utils.conf
/etc/amazon/efs/efs-utils.crt
/usr/bin/amazon-efs-mount-watchdog
/usr/lib/systemd/system/amazon-efs-mount-watchdog.service
/usr/sbin/mount.efs
/usr/share/doc/efs-utils
/usr/share/doc/efs-utils/CONTRIBUTING.md
/usr/share/doc/efs-utils/README.md
/usr/share/licenses/efs-utils
/usr/share/licenses/efs-utils/LICENSE
/usr/share/man/man8/mount.efs.8.gz
/var/log/amazon/efs

Let’s get the watchdog running so we have that ready later. The watchdog helps to build and tear down the encrypted connection when you mount and unmount an EFS volume:

$ sudo systemctl enable --now amazon-efs-mount-watchdog.service
Created symlink /etc/systemd/system/multi-user.target.wants/amazon-efs-mount-watchdog.service → /usr/lib/systemd/system/amazon-efs-mount-watchdog.service.
$ systemctl status amazon-efs-mount-watchdog.service
● amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog
 Loaded: loaded (/usr/lib/systemd/system/amazon-efs-mount-watchdog.service; enabled; preset: disabled)
 Drop-In: /usr/lib/systemd/system/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Wed 2023-09-13 18:43:46 UTC; 5s ago
 Main PID: 1258 (amazon-efs-moun)
 Tasks: 1 (limit: 4385)
 Memory: 13.3M
 CPU: 76ms
 CGroup: /system.slice/amazon-efs-mount-watchdog.service
 └─1258 /usr/bin/python3 /usr/bin/amazon-efs-mount-watchdog

Sep 13 18:43:46 ip-172-31-2-38.us-east-2.compute.internal systemd[1]: Started amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog.

Setting up an EFS volume #

Start by going over to the EFS console and do the following:

Click File systems in the left navigation bar
Click the orange Create file system button at the top right

A modal appears with a box for the volume name and a VPC selection. Select an easy to remember name (I’m using testing-efs-for-blog-post) and select a VPC. If you’re not sure what a VPC is or which one to use, use the default VPC since that’s likely where your instance landed as well.
Click Create.

There’s a delay while the filesystem initializes and you should see the filesystem show Available with a green check mark after about 30 seconds. Click on the filesystem you just created from the list and you’ll see the details page for the filesystem.

Security setup #

EFS volumes come online with the default security group attached and that’s not helpful. From the EFS filesystem details page, click the Network tab and then click Manage.

For each availability zone, go to the Security groups column and add the security group that your instance came up with in the first step. In my case, I accepted the defaults from EC2 and ended up with a launch-wizard-1 security group. Remove the default security group from each. Click Save.

Mounting time #

You should still be on the filesystem details page from the previous step. Click Attach at the top right and a modal will appear with mount instructions. The first option should use the EFS mount helper!

For me, it looks like sudo mount -t efs -o tls fs-0baabc62763375bb1:/ efs

Go back to your Fedora instance, create a mount point, and create the volume:

$ sudo mkdir /mnt/efs
$ sudo mount -t efs -o tls fs-0baabc62763375bb1:/ /mnt/efs
$ df -hT | grep efs
127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /mnt/efs

We did it! 🎉

We see 127.0.0.1 here because efs-utils uses stunnel to handle the encryption between your instance and the EFS storage system.

The disk was mounted by root, so we can add a -o user=fedora to give our Fedora user permissions to write files:

$ umount /mnt/efs
$ sudo mount -t efs -o user=fedora,tls fs-0baabc62763375bb1:/ /mnt/efs
$ touch /mnt/efs/test2.txt
$ stat /mnt/efs/test2.txt
 File: /mnt/efs/test2.txt
 Size: 0 	Blocks: 8 IO Block: 1048576 regular empty file
Device: 0,54	Inode: 17657675890899444015 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ fedora) Gid: ( 1000/ fedora)
Context: system_u:object_r:nfs_t:s0
Access: 2023-09-13 19:14:23.308000000 +0000
Modify: 2023-09-13 19:14:23.308000000 +0000
Change: 2023-09-13 19:14:23.308000000 +0000
 Birth: -

Also, efs-utils uses encrypted communication by default, which is great. There may be some situations where you don’t need encrypted communications or you don’t want the overhead. In that case, drop the -o tls option from the mount command and you’ll mount the volume unencrypted.

$ sudo umount /mnt/efs
$ sudo mount -t efs -o user=fedora fs-0baabc62763375bb1:/ /mnt/efs
$ df -hT | grep efs
fs-0baabc62763375bb1.efs.us-east-2.amazonaws.com:/ nfs4 8.0E 0 8.0E 0% /mnt/efs

Extra credit #

You can get fancy with access points that allow you to carve up your EFS storage and only let certain instances mount certain parts of the filesystem. So instance A might only be able to mount /files/hr while instance B can only mount /documents.

It would also be a good idea to take an inventory of your security groups and ensure the least amount of instances can reach your EFS volume as possible. Much of the work I did in this post was just for testing. A good plan might be to make a security group for your EFS volume and only allow inbound traffic from security groups which should access it. That would allow you to gather up all of your instances into different security groups and limit access.

Also, be aware of the EFS pricing! 💸

You are billed not only for how much storage you use, but also on requests. Different requests are priced differently depending on access frequency. Backups are also enabled by default at $0.05/GB-month!

Why Ohio? I’m mainly doing it to irritate Corey Quinn. 🤭 Any region you prefer should be fine. ↩︎

Car buying guide

major@mhtx.net (Major Hayden) — Mon, 04 Sep 2023 00:00:00 +0000

I love optimizing nearly everything in my life. Sometimes it means saving money. Other times it means squeezing every bit of performance out of a server.

But let’s try optimizing something I’ve never done on this blog before: Buying a car.

Although most of this information applies best to people in the USA, there are several things I’ve learned over the years that might benefit people in other places. Car purchases are often the second largest purchase for most Americans after buying a house. Why not optimize it as much as possible?

My family is in the market for new vehicle and I’ve immersed myself in learning more about the whole process. There are plenty of legal issues in play that many buyers don’t know about and there are tons of ways to walk into a dealership fully prepared.

Without further ado, I’ll share what I’ve learned with you!

Shopping #

Shopping and buying are two different things. You must keep them separate. If you’re not sure on the exact model of car you want, go shopping and don’t commit to buy anything.

If you have an idea of a particular car you want, go to Google and search for 5-10 competitors to that car. It’s as easy as searching “top competitors Camry” to find other cars that compete with a Toyota Camry. Go to those dealers and take a good look at the cars to see if they have the features you want. Test drive each and see if you still like them.

The really important thing here is to separate shopping from buying.

Search for prices #

Once you narrow your search down a bit, start to compare prices between dealers for different trim levels. I love using CarEdge for this since you can examine dealer inventory across the country and see how dealer prices stack up against each other.

Also check how long the vehicle has been sitting on the lot. CarEdge offers this data and it can give you an idea of how desperate a dealer is to get rid of a particular vehicle. If the total supply of a model is really high and the vehicle has been sitting on the lot longer than 90 days, you have the ability to negotiate for that car. Cars with a very low supply or cars that just arrived to the lot will likely be priced higher and dealers are less likely to budge on price.

For used cars, this is even more important since you don’t have an MSRP to work with as you do for new cars. Comparing prices for used cars is critical.

Keep in mind that some cars are in higher demand in some areas than others, too. You might be able to drive a few hours and save quite a bit. A friend of mine drove from Texas to Colorado to buy a car and saved $3,500.

Watch out for arbitrary markups! All dealers in the US are required to display a Monroney label and this shows the manufacturer’s suggested retail price. Dealers might add an addendum sticker somewhere else that show accessories they added. Sometimes these stickers show arbitrary markups from the dealer.

Since COVID, many dealers have been stacking massive fees on top of car purchases. Some of these fees exceed $10,000, $25,000 or more! This should be a massive red flag from the start of a negotiation with any dealer. If they aren’t willing to budge from their arbitrary markups, look for another dealer. 🚩

You found the car you want #

Awesome! 🥳

Whether it’s new or used, take the VIN number and dig up information about the car. You can learn a lot from just a quick Google search from the VIN!

Other than that, CarEdge has some good tools for digging up data on individual cars without too much expense. CarFax is the gold standard used and it offers some fairly inexpensive options.

Some might be saying: “Isn’t this a waste for a brand new car?” No, it’s not.

I was once buying a pickup truck that was advertised as one model year, but the truck was actually a year older. The dealer even switched the paper tags attached to the keys so I wouldn’t notice. I didn’t notice this until months later when I found it buried in my paperwork. 😡

There have also been situations where stolen cars are sitting on dealer lots. 😱

Remember, this is a big investment. Spending $20 on a CarFax report as you purchase a $30,000 car should be worth it.

Financing #

Take care of your financing before visiting a dealer. Local credit unions are often great for this but they’ve been under a squeeze lately with an increase in reposessions. Work with the credit union to get a good rate for the type of car you want.

Some people have had good luck financing through a dealership. However, getting financing set up ahead of time gives you the upper hand with financing negotiations. For example, when you sit down with in the finance office, you could say: “I have 5.9% through my local credit union. If you can beat that, I’ll finance with you.”

If you choose to go through financing with the dealer, here’s what I recommend:

Do not allow the dealer to run your credit report until the last minute. If you let them run it, they know that you’ll get a hard hit on your credit report and you’re less likely to look at other dealers. You want to leave your options open. Don’t let them run your credit until you are 100% sure you’re ready to buy from them.
Ask to see the “call sheet” and see what the bank offered the dealer for financing (the “buy rate”). It’s very likely that the dealer gets one rate from the bank and then marks it up for you. For example, the dealer might say “Oh, your rate is 8%.” Then you ask for the call sheet or the buy rate and see that they got 5.9% from the bank. You’re getting a huge markup. That’s another negotiation point.
If the dealer says they can lower your interest rate if you buy an add-on from them, such as extended warranty, STOP IMMEDIATELY. 🚨 This is called “tied selling” and is almost always illegal in the USA. This is a good moment to stop and re-evaluate the whole transaction.
You are not required to buy any of the add-ons that the finance manager offers you. Extended warranties, dent and ding protection, and tire/wheel protection are common items. There’s no requirement to purchase these. If you do decide to purchase one, be sure they can show you the actual amount that it will cost you. Don’t let them tell you how much it adds to your monthly payment. That allows them to hide the cost of certain add-ons.
You are required to pay reasonable fees for tax, title, and license. Sometimes documentation fees are rolled up into this, too. Every state is a little different on how much these cost, but you can Google “tax title and license” for your state and get a good estimation.
Demand to see the “out the door price.” If a dealer asks how much you can afford per month, don’t answer. You are interested in the full price of the car plus fees. Most states require that this comes out on a single sheet of paper with each expense clearly labeled with what you will owe for the car. If you tell a dealer “I can’t affort more than $500 per month”, then they will tinker with various parts of the deal to ensure you pay more in the long term without exceeding $500 per month. If the dealer won’t budge, keep asking them “If I was getting you a cashier’s check today, how much needs to be on the check?” They will eventually get the idea.

Dealers make the most money by far not on the lot itself, but in the finance office. Don’t get swindled there.

Get copies and read them #

You are entitled to a copy of everything that shows up in the finance office. Don’t get up from the chair until you have a copy of everything and you’ve examined each page.

Dealers commonly adjust numbers or conveniently leave out “We Owe” sheets (see the next section) in the finance office. Sometimes it’s an honest mistake. Sometimes it’s not.

The dreaded “We Owe” #

If a dealer doesn’t have something in stock that they promised you, such as an accessory or add-on, ensure it lands on the “We Owe” sheet in your paperwork.

There should be page somewhere in your sale paperwork that shows anything that the dealer owes you. In many states, this sheet must be in your paperwork even if it’s blank or zeroed out!

If a dealer promised you something and it didn’t land on the “We Owe” sheet, stop immediately and ask for that to be corrected right then.

Trade-ins #

Any time you trade in a vehicle, make it a different transaction. Allowing a dealer to add your trade-in with the current deal for purchase allows them to hide money for themselves in the deal.

First, get multiple offers from various sites that buy cars all day long. I recommend Carvana, Driveway, and Vroom. They will give you an immediate estimate online with a little bit of information. The CarEdge site I mentioned earlier also has some tools that allow you to look up your car in the Black Book, which is what dealers use to appraise cars.

Next, when you go to the dealer and they ask if you’re trading in a car, tell them you haven’t decided yet. Your best bet is to act as if you want them to talk you into it. Either way, keep the trade-in as a separate transaction.

When it comes time to talk about your trade-in, you should already have an out the door price on your car purchase. Scroll up if you’re not sure about this. 😉

Let the dealer know you have some other offers on your trade and let them know you’ll do the trade there if they can beat the offers. If they offer to beat the other deals, that’s awesome! That’s less work for you!

If they can’t, don’t worry. You can handle the trade-in separately with those other companies.

Buying online #

Finally, you can buy a car almost entirely online these days. Carvana, Driveway, and Vroom offer a fully online experience, but traditional dealers often have salespeople focused on internet deals.

This is a good way to sort out dealers who want to work with you and those that don’t. Let dealers know that:

You know what vehicle you want.
You’re comparing the offers from multiple dealers.
You’re looking to purchase within the next few weeks.
You want an out the door price on the vehicle so you know how much to get on the cashier’s check.

If they’re willing to deal with you via email or phone, that’s great! If not, there’s plenty of other dealers out there.

Further learning #

I’ve learned a lot from several YouTube channels over the years. Here are my favorites:

Car Questions Answered: Brandon runs a small used car dealership in North Carolina and gives lots of insights on what is happening the used and new car markets. He has lots of good advice on when to buy a car and which cars are the ones to avoid at certain times. If you ever wanted to go behind the scenes to see how a small used car dealership works, his channel is great.
Deshone The Auto Advisor: Deshone has lots of helpful short videos. He does offer a membership program that comes with a fee, but his short videos cover a ton of car buying and leasing recommendations. If you’re interested in leasing a car, be sure to watch some of his leasing videos.
CarEdge: CarEdge offers tons of services to help with car buying including 1:1 coaching once you have a deal sheet from a dealer. However, they also have tons of videos that explain how to buy a car effectively. They do role playing for finance managers and salespeople that highlight certain areas where buyers usually lose. Their role play videos are highly recommended.
Lucky Lopez: Lucky has tons of insight from a dealer perspective and he follows lots of trends around pricing, supply, and reposessions. Most of his content might be too much for car buyers, but it’s good information to know.
Chevy Dude: The Chevy Dude used to work for multiple dealers but now it running his own. He shares lots of sales tricks and secrets that help you prepare for making your next deal on a car – new or used.

Did I miss something? #

Let me know if I missed something and I’ll come back and edit this post! Just contact me via any of the methods below in the author block. ️⬇️

Fixing a ghost database migration failure

major@mhtx.net (Major Hayden) — Thu, 31 Aug 2023 00:00:00 +0000

I love learning about the behind the scenes aspects of just about everything. I do ham radio, I self-host lots of my personal infrastructure, and I’ve been learning more about the math behind the stock market for the last year or two.

That led me to start a blog on Ghost to share my findings with others. I started Theta Nerd¹ earlier this summer.

My deployment looked great when I started! Everything was automatically updated with watchtower and running with docker-compose on Fedora CoreOS. (Click these links to read the posts on both topics!)

However, I woke up one morning to my monitoring going off and my site was down. 😱

Why is the site down? #

Anyone who has worked in IT knows this sinking feeling. Something is down, you don’t know why, and you suspect the worst possible scenarios.

The instance hosting the blog was online and responsive, so I started digging into the logs with docker-compose logs. I suddenly found a wall of text in the logs for the Ghost container:

[2023-08-03 11:10:16] INFO Adding members.email_disabled column
[2023-08-03 11:10:16] INFO Setting email_disabled to true for all members that have their email on the suppression list
[2023-08-03 11:10:16] INFO Setting nullable: stripe_products.product_id
[2023-08-03 11:10:16] INFO Adding table: donation_payment_events
[2023-08-03 11:10:16] INFO Rolling back: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible..
[2023-08-03 11:10:16] INFO Dropping table: donation_payment_events
[2023-08-03 11:10:16] INFO Dropping nullable: stripe_products.product_id with foreign keys disabled
[2023-08-03 11:10:16] INFO Setting email_disabled to false for all members
[2023-08-03 11:10:16] INFO Removing members.email_disabled column
[2023-08-03 11:10:16] INFO Rollback was successful.
[2023-08-03 11:10:16] ERROR alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.

alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
{"config":{"transaction":false},"name":"2023-07-27-11-47-49-create-donation-events.js"}
"Error occurred while executing the following migration: 2023-07-27-11-47-49-create-donation-events.js"
Error ID:
 300
Error Code: 
 ER_FK_INCOMPATIBLE_COLUMNS
----------------------------------------
Error: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
 at /var/lib/ghost/versions/5.57.2/node_modules/knex-migrator/lib/index.js:1032:19
 at Packet.asError (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packets/packet.js:728:17)
 at Query.execute (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/commands/command.js:29:26)
 at Connection.handlePacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:478:34)
 at PacketParser.onPacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:97:12)
 at PacketParser.executeStart (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packet_parser.js:75:16)
 at Socket.<anonymous> (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:104:25)
 at Socket.emit (node:events:513:28)
 at addChunk (node:internal/streams/readable:315:12)
 at readableAddChunk (node:internal/streams/readable:289:9)
 at Socket.Readable.push (node:internal/streams/readable:228:10)
 at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

Ah, so a failed database migration in the upgrade to 5.57.2 is the culprit! 👏

I brought the site back online quickly by changing the container version for Ghost back to the previous version (5.55.2).

Why did the database migration fail? #

The error message from above boils down to this:

Error: alter table `donation_payment_events` add constraint 
`donation_payment_events_member_id_foreign` foreign key (`member_id`)
references `members` (`id`) on delete SET NULL - Referencing column
'member_id' and referenced column 'id' in foreign key constraint 
'donation_payment_events_member_id_foreign' are incompatible.

Adjusting the donation_payment_events.member_id column to be a foreign key of members.id is failing because they are incompatible types. However, as I examined both tables, both were regular varchar(24) columns without anything special attached to them:

mysql> describe members;
+------------------------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------------+---------------+------+-----+---------+-------+
| id | varchar(24) | NO | PRI | NULL | |
| uuid | varchar(36) | YES | UNI | NULL | |
| email | varchar(191) | NO | UNI | NULL | |
| status | varchar(50) | NO | | free | |
| name | varchar(191) | YES | | NULL | |
| expertise | varchar(191) | YES | | NULL | |
| note | varchar(2000) | YES | | NULL | |
| geolocation | varchar(2000) | YES | | NULL | |
| enable_comment_notifications | tinyint(1) | NO | | 1 | |
| email_count | int unsigned | NO | | 0 | |
| email_opened_count | int unsigned | NO | | 0 | |
| email_open_rate | int unsigned | YES | MUL | NULL | |
| last_seen_at | datetime | YES | | NULL | |
| last_commented_at | datetime | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| created_by | varchar(24) | NO | | NULL | |
| updated_at | datetime | YES | | NULL | |
| updated_by | varchar(24) | YES | | NULL | |
+------------------------------+---------------+------+-----+---------+-------+
18 rows in set (0.00 sec)

Going upstream #

I went to Ghost’s GitHub repository and opened an issue with as much data as I can find.

One of the first replies mentioned something about database collations. Long story short, collations describe how databases handle sorting and comparing data for different languages. Comparing some languages to other languages can be particularly challenging and this can lead to problems.

I made a switch from MariaDB to MySQL recently for the blog. Could that be related?

More searching #

I figured that I wasn’t the first one to stumble into this problem, and sure enough – I wasn’t! There’s a great blog post about a broken migration from MySQL 5 to 8 with Ghost.

In short, it required several steps to fix it:

Stop the Ghost container
Back up the database first (always a good idea)
Do a quick find/replace on the dumped database to change the collations
Drop the ghost database from the database 😱
Import the database back into MySQL
Start Ghost again

Dropping databases always makes me pause, but that’s what backups are for! 😉

How I fixed it #

In my case, my MySQL container is called ghostmysql and my Ghost database is ghostdb. Then I made a backup of the database using mysqldump:

sudo docker-compose exec ghostmysql mysqldump \
 -u root -psuper-secret-password ghostdb > backup-ghost-db.sql

Next, I copied the SQL file to another directory just in case I accidentally deleted this backup with an errant command.

cp backup-ghost-db.sql ../

Then I made a copy of the SQL file in the current directory and ran the find and replace on that copy. This changes the collations from the wrong one, utf8mb4_general_ci, to the right one, utf8mb4_0900_ai_ci²:

cp backup-ghost-db.sql backup-ghost-db-new.sql
sed -i 's/utf8mb4_general_ci/utf8mb4_0900_ai_ci/g' \
 backup-ghost-db-new.sql

Now I have the collations right for importing the database back into MySQL. But first, I have to drop the existing database. This is a good time to double check your backups!

sudo docker-compose exec ghostmysql mysql -u root \
 -psuper-secret-password
mysql> DROP DATABASE ghostdb;

Now we can import the modified backup:

cat backup-ghost-db-new.sql | sudo docker-compose exec -T \
 ghostmysql mysql -u root -psuper-secret-password ghostdb

Start all the containers:

sudo docker-compose up -d

Ghost was back online with the older version and everything looked good! I updated my docker-compose.yaml back to use latest for the Ghost version and ran sudo docker-compose up -d once more.

Within seconds, the new container image was in place and the container was running! Both migrations completed in seconds and the blog was back online with the newest version. 🎉

Theta is one of many financial Greeks that measure certain aspects of options contracts in the market. It’s also a letter in the Greek alphabet. ↩︎
The default collation in MySQL 8 is utf8mb4_0900_ai_ci. ↩︎

Open source contributions: Just do it

major@mhtx.net (Major Hayden) — Wed, 16 Aug 2023 00:00:00 +0000

I had a great time on the Fedora Podcast yesterday to talk about Fedora cloud! We talked about all kinds of Fedora-related topics, but a couple of questions came up around how to contribute, especially when there’s not a lot of structure in place for a particular type of contributions. Here’s the full video if you’re interested:

That made me think about a post that deserves to be written: How do you get started with open source contributions in a new project? 🤔

My answer is pretty simple: Just do it. 🚀

Just do what? #

In the late 1980’s, one of Nike’s ad agencies came up with the phrase as a way to push through uncertainty. I was pretty young when this campaign started, but the general idea was this:

Anyone can achieve what they want
Stop worrying about whether you can actually do something
Try something new
Just do it

Simple, right?

This works for open source contributions, too. I often have conversations with people inside and outside of work where they identify a problem or an improvement in an open source project. My customary response is “Let’s go upstream and make this better!”

However, what I hear back most often is “I don’t know how.” This is where the whole just do it part comes in.

I found a bug #

Nearly every open source project wants to know about bugs that users experience. Start by finding out the best way to communicate with the people working on a particular project.

For projects on GitHub or GitLab, you can open up an issue and describe your problem. Some repositories have a template generated for bugs that ask you several important questions, so be sure to follow those templates. If there isn’t a template, I usually follow this format:

What happened that was unexpected?
What were you doing right before that unexpected event happened?
What did you expect to happen instead?
What else is nearby in the environment that might have an impact?
- For example, the versions of Python might be important for Python-based projects.
What log files or other diagnostic materials exist?

What’s the goal? We want to give maintainers enough information for a quick diagnosis in the best case. If it’s not obvious, then they need enough information to try to reproduce it on their own machine for debugging.

Maintainers might come back with additional questions about your environment or the events just before the bug occurred. Be sure to respond in a timely way while the information is top of mind for them.

Always remember that these maintainers are real people who are likely not being paid for their work. Assume the best of intentions (unless proven otherwise) and stay focused on the solution. There might always be the chance that the maintainers are not interested in your use case and might not be interested in solving it.

That leads me to the next step.

I found a bug and I want to fix it #

Start by opening an issue or a bug report first (see the previous section).

This ensures that maintainers get a full picture of the problem you’re trying to solve. Also, I’ve had maintainers immediately reply and tell me that it’s a known issue already being solved in another issue. That could save you some work.

If you have a patch that fixes the issue, go through the following steps before submitting the fix upstream:

Ensure your fix references the issue or bug report that you opened
Use a very clear first line in your commit message, such as parser: Fix emoji handling in YAML rather than Fix YAML bug
Include a very brief explanation of the bug you’re fixing in the commit message
Extra credit: Add or update existing tests so they catch the bug you just found
Extra credit: Add or update the project documentation for your change if necessary

These extra credit items often make it easier to review your patch. Maintainers love extra test coverage, too.

Submit your change in a pull request or merge request and watch for updates. Be patient with replies from the maintainers, but be timely in your replies. Remember that your use case might be an edge case for the upstream project and you might need to explain your fix (or the original bug) in more detail.

I want to improve something #

Improving an open source project could involve several things, such as:

Enhancing by adding a new feature
Optimizing an existing feature
Creating documentation
Building integrations

I strongly recommend opening an issue first with the project maintainers to explain your enhancement. These Requests for Enhancements, or RFEs, should include several things:

Your use case that made you think of the enhancement in the first place
What you plan to add, substract, or change
How the changes might affect different users, especially as they upgrade from older versions
How the changes might affect testing or release processes
Any changes in dependencies required

Before going down the road of enhancements, always bring up these ideas with the maintainers first. You want to ensure that your ideal changes are aligned with the future goals of the project. In addition, maintainers will want to better understand your use case.

Remember that an enhancement almost always requires additional work from maintainers. Every new use case means more work to ensure the project still functions. That’s why it’s critical to share your use case and have a good plan for testing and documentation.

Getting involved #

Whenever I find an open source project that I’d like to get involved with, I start looking around for several things:

What do they use for informal asynchronous chat? IRC? Matrix? Slack? Something else? I join the chat, introduce myself, and get an idea for how they interact. Some groups are very chatty and informal while others are much more formal and regimented.
Where do they have detailed discussions? Many projects have detailed discussions in their issues/bugs or in places like GitHub’s discussions. Others use old school mailing lists. Some groups have regular meetings where anyone can add agenda items for discussions. If I need to talk about something a bit more long form and I expect some back and forth on it, I look for this avenue.
What requirements exist for contributors? Some projects require that contributors sign a CLA or some other sort of agreement. Make sure that any CLAs you sign are approved by your employer (if applicable). You might need an account on a system that you don’t have, so check for that as well.

From there, I take the just do it mentality and go for it. The worst thing you’ll be told is “No”. If that happens, take a step back, see if there’s another way to approach it, and try again.

Remember one thing most of all: avoid taking anything personally. All of us have our bad days and some people have personalities that might be totally incompatible with yours (and most people in general). 🤭

Major Hayden

Repairing 4Runner skid plate bolts

Backstory #

Root cause #

Ingredients #

Fix the bolt holes #

Replace the skid plate #

Prevention #

Spell check in multiple languages with Firefox

Installing languages #

Enable the language #

My meeting hacks

Use headphones or earbuds #

Background music #

Ask about taking notes #

Decline that meeting! #

Rub some AI on it

AI is not valuable alone #

Work backwards from the experience #

Remember the human side #

So what do we do? #

AMD GPU missing from btop

The problem #

The solution #

Running ollama with an AMD Radeon 6600 XT

It’s so slow 🐌 #

The fix #

Giving it another try #

Jellyfin fatal player error

Checking the logs #

Checking the browser again #

Jellyfin deployment #

Users and groups #

Redirect local ports with firewalld

Old-school iptables methods #

Why consider firewalld? #

Port redirections in firewalld #

Extra credit #

amazon-ec2-utils in Fedora

The problem #

udev rules to the rescue #

Fix big cursors in Java applications in Wayland

Fixing big cursors #

Desktop file #

systemd does everything 😆 #

cloud-init and dhcpcd

What’s new with dhcpcd? #

But I use NetworkManager #

But I use systemd-networkd #

How can I get it right now? #

Texas Linux Fest 2024 recap 🤠

Containers talk #

Tech career talk #

Other talks #

Around the exhibit hall #

Roll your own static blog analytics

Why do you need analytics? #

What are the ingredients? #

A static blog in a container? #

Caddy logs #

Enabling analytics #

Serving the blog #

What about new posts? #

Connect Caddy to Porkbun

DNS validation #

Adding Porkbun support to Caddy #

Automated Caddy builds with updates #

Connecting to Porkbun #

Renewals #

Further reading #

Linux on the AMD ThinkPad Z13 G2

Power management #

Touchpad #

Display #

Audio #

Everything else #

Dark mode in Sway

GTK applications #

QT applications #

Alternate dark mode based on time #