Major Haydenhttps://major.io/Recent content on Major HaydenHugoenmajor@mhtx.net (Major Hayden)major@mhtx.net (Major Hayden)All content licensed [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) πŸ’œWed, 17 Jul 2024 16:44:43 +0000Jellyfin fatal player errorhttps://major.io/p/jellyfin-fatal-player-error/Tue, 02 Jul 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/jellyfin-fatal-player-error/<p>Plex has been a mainstay for serving up media at home but it seems to have changed lately towards a more and more commercial offering. A friend recommended <a href="https://jellyfin.org/" target="_blank" rel="noreferrer">Jellyfin</a> and I deployed it on my Synology NAS in a Docker container.</p> <p>I did a few quick tests in a web browser and everything looked good. But then my Jellyfin android app told me:</p> <blockquote> <p>Playback failed due to a fatal player error</p> </blockquote> <p>Everything looked fine in the browser, so it was time to dig in.</p> <h2 id="checking-the-logs" class="relative group">Checking the logs <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#checking-the-logs" aria-label="Anchor">#</a></span></h2><p>I opened up an ssh connection on the Synology to check the logs and found something unhelpful:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plain" data-lang="plain"><span class="line"><span class="cl">Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 1 </span></span></code></pre></div><p>Running a few searches led me down rabbit holes to plenty of GitHub issues. None of them fixed the issue.</p> <h2 id="checking-the-browser-again" class="relative group">Checking the browser again <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#checking-the-browser-again" aria-label="Anchor">#</a></span></h2><p>I went through a few different videos from the Synology and played each. They all looked fine in Firefox until I reached one that seemed to stutter. The frame rate looked as if at least half of the frames were bring dropped.</p> <p>That particular video was in 4K with a high bit rate. Back on the synology, the CPU usage was through the roof.</p> <p>I configured graphics acceleration when I deployed Jellyfin. Perhaps it wasn&rsquo;t working?</p> <h2 id="jellyfin-deployment" class="relative group">Jellyfin deployment <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#jellyfin-deployment" aria-label="Anchor">#</a></span></h2><p>I deployed Jellyfin using the upstream guides with docker-compose:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">jellyfin</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/jellyfin/jellyfin:latest</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">jellyfin</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="m">1026</span><span class="p">:</span><span class="m">100</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">network_mode</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;host&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">devices</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">/dev/dri</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># removed</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;unless-stopped&#34;</span><span class="w"> </span></span></span></code></pre></div><p>One of the GitHub issues I stumbled upon suggested being specific about the video devices that are mounted inside the container.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> ls -al /dev/dri </span></span><span class="line"><span class="cl"><span class="go">total 0 </span></span></span><span class="line"><span class="cl"><span class="go">drwxr-xr-x 2 root root 80 Jun 10 20:23 . </span></span></span><span class="line"><span class="cl"><span class="go">drwxr-xr-x 12 root root 14140 Jun 10 20:24 .. </span></span></span><span class="line"><span class="cl"><span class="go">crw------- 1 root root 226, 0 Jun 10 20:24 card0 </span></span></span><span class="line"><span class="cl"><span class="go">crw-rw---- 1 root videodriver 226, 128 Jun 10 20:24 renderD128 </span></span></span></code></pre></div><p>I adjusted the deployment in <code>docker-compose.yaml</code> and tried again:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">jellyfin</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/jellyfin/jellyfin:latest</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">jellyfin</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="m">1026</span><span class="p">:</span><span class="m">100</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">network_mode</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;host&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">devices</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">/dev/dri/renderD128:/dev/dri/renderD128</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">/dev/dri/card0:/dev/dri/card0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># removed</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;unless-stopped&#34;</span><span class="w"> </span></span></span></code></pre></div><p>I redeployed jellyfin:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> docker-compose up -d jellyfin </span></span></code></pre></div><p>The Android app still had the fatal player error.</p> <h2 id="users-and-groups" class="relative group">Users and groups <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#users-and-groups" aria-label="Anchor">#</a></span></h2><p>Most of my Synology containers use the uid/gid pair of <code>1026:100</code> so allow them to read and write to my storage volume. The <code>/dev/dri/renderD128</code> is owned by the <code>videodriver</code> group:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> grep videodriver /etc/group </span></span><span class="line"><span class="cl"><span class="go">videodriver::937:PlexMediaServer </span></span></span></code></pre></div><p>This likely came from a time when I installed Plex on Synology via one of the Synology applications rather than from a container. <em>(I&rsquo;m not sure, but that&rsquo;s my guess.)</em></p> <p>I added that group to the container:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">jellyfin</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/jellyfin/jellyfin:latest</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">jellyfin</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="m">1026</span><span class="p">:</span><span class="m">100</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">network_mode</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;host&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group_add</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">&#34;937&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">devices</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">/dev/dri/renderD128:/dev/dri/renderD128</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">/dev/dri/card0:/dev/dri/card0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># removed</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;unless-stopped&#34;</span><span class="w"> </span></span></span></code></pre></div><p>After redeploying the container, the Android app worked just fine! Also, the video stuttering disappeared when viewing the 4K video from the browser. πŸŽ‰</p>Redirect local ports with firewalldhttps://major.io/p/firewalld-port-redirection/Fri, 28 Jun 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/firewalld-port-redirection/<p>Linux networking and firewalls give us plenty of options for redirecting traffic from one port to another. We can allow people outside our home to reach a web server we run in our internal network. That&rsquo;s called destination NAT, ot <a href="https://en.wikipedia.org/wiki/Network_address_translation#DNAT" target="_blank" rel="noreferrer">DNAT</a>.</p> <p>You can also redirect traffic to different ports on the same host. For example, if you have a daemon listening on port 3000, but you want people to reach that service on port 80, you can redirect traffic from 80 to 3000 on the same host (without network address translation).</p> <p>But how do we do this with <a href="https://firewalld.org/" target="_blank" rel="noreferrer">firewalld</a>? πŸ€”</p> <h2 id="old-school-iptables-methods" class="relative group">Old-school iptables methods <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#old-school-iptables-methods" aria-label="Anchor">#</a></span></h2><p>Let&rsquo;s say you have a service running on port 3000 and you want to expose it to other computers on your same network as port 80. With iptables, you would typically start by enabling IP forwarding:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo sysctl -w net.ipv4.ip_forward=1 </span></span></span></code></pre></div><p>Add two iptables rules to handle packets coming in from the outside as well as any locally generated packets:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">#</span> Handle locally-generated packets on the same machine. </span></span><span class="line"><span class="cl"><span class="go">sudo iptables -t nat -A PREROUTING -s 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to 3000` </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="gp">#</span> Handle packets coming from outside the current machine. </span></span><span class="line"><span class="cl"><span class="go">sudo iptables -t nat -A OUTPUT -s 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to 3000` </span></span></span></code></pre></div><p>There&rsquo;s a weird situation that happens on certain machines with certain network configurations where packets are not properly routed when they are destined for the local network adapter. To fix that, set one more sysctl configuration:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo sysctl -w net.ipv4.conf.all.route_localnet=1 </span></span></span></code></pre></div><p>Remember to make these sysctl configurations permanent:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo mkdir /etc/sysctl.conf.d/ </span></span></span><span class="line"><span class="cl"><span class="go">echo &#34;net.ipv4.ip_forward=1&#34; | sudo tee &gt;&gt; /etc/sysctl.conf.d/redirect.conf </span></span></span><span class="line"><span class="cl"><span class="go">echo &#34;net.ipv4.conf.all.route_localnet&#34; | sudo tee &gt;&gt; /etc/sysctl.conf.d/redirect.conf </span></span></span></code></pre></div><h2 id="why-consider-firewalld" class="relative group">Why consider firewalld? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#why-consider-firewalld" aria-label="Anchor">#</a></span></h2><p>I like firewalld because I can manage lots of settings for different firewall zones and allow access from one zone to another. It also allows me to put certain interfaces in trusted zones so they automatically get more access.</p> <p>Another nice aspect about firewalld is that it supports iptables and nftables backends. You don&rsquo;t have to think about the differences between the backends. All of that is taken care of for you.</p> <h2 id="port-redirections-in-firewalld" class="relative group">Port redirections in firewalld <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#port-redirections-in-firewalld" aria-label="Anchor">#</a></span></h2><p>Let&rsquo;s start by checking our default firewalld zone:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo firewall-cmd --list-all </span></span><span class="line"><span class="cl"><span class="go">FedoraServer (default, active) </span></span></span><span class="line"><span class="cl"><span class="go"> target: default </span></span></span><span class="line"><span class="cl"><span class="go"> ingress-priority: 0 </span></span></span><span class="line"><span class="cl"><span class="go"> egress-priority: 0 </span></span></span><span class="line"><span class="cl"><span class="go"> icmp-block-inversion: no </span></span></span><span class="line"><span class="cl"><span class="go"> interfaces: bond0 eno1 eno2 </span></span></span><span class="line"><span class="cl"><span class="go"> sources: </span></span></span><span class="line"><span class="cl"><span class="go"> services: dhcpv6-client http https </span></span></span><span class="line"><span class="cl"><span class="go"> ports: 51820/udp </span></span></span><span class="line"><span class="cl"><span class="go"> protocols: </span></span></span><span class="line"><span class="cl"><span class="go"> forward: yes </span></span></span><span class="line"><span class="cl"><span class="go"> masquerade: yes </span></span></span><span class="line"><span class="cl"><span class="go"> forward-ports: </span></span></span><span class="line"><span class="cl"><span class="go"> source-ports: </span></span></span><span class="line"><span class="cl"><span class="go"> icmp-blocks: </span></span></span><span class="line"><span class="cl"><span class="go"> rich rules: </span></span></span></code></pre></div><p>This output shows that my external network interfaces are attached to the zone and forwarding is already on in my case. If you see <code>forward: no</code> here, just run this command:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo firewall-cmd --add-forward </span></span><span class="line"><span class="cl"><span class="go">success </span></span></span></code></pre></div><p>Now firewalld will manage your <code>forwarding</code> sysctl variables for you on these interfaces. That&rsquo;s handy. πŸ˜‰</p> <p>Next, let&rsquo;s get the redirect working. We want to take external packets on port 80 and send them to 3000: on the local machine.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo firewall-cmd <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span><span class="go"> --add-forward-port=port=80:proto=tcp:toport=3000:toaddr=127.0.0.1 </span></span></span><span class="line"><span class="cl"><span class="go">success </span></span></span></code></pre></div><p>In this command, we told firewalld to take 80/tcp from the outside and send it to port 3000 on the local host (127.0.0.1). Let&rsquo;s double check our current configuration:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo firewall-cmd --list-all </span></span><span class="line"><span class="cl"><span class="go">FedoraServer (default, active) </span></span></span><span class="line"><span class="cl"><span class="go"> target: default </span></span></span><span class="line"><span class="cl"><span class="go"> ingress-priority: 0 </span></span></span><span class="line"><span class="cl"><span class="go"> egress-priority: 0 </span></span></span><span class="line"><span class="cl"><span class="go"> icmp-block-inversion: no </span></span></span><span class="line"><span class="cl"><span class="go"> interfaces: bond0 eno1 eno2 </span></span></span><span class="line"><span class="cl"><span class="go"> sources: </span></span></span><span class="line"><span class="cl"><span class="go"> services: dhcpv6-client http https </span></span></span><span class="line"><span class="cl"><span class="go"> ports: 51820/udp </span></span></span><span class="line"><span class="cl"><span class="go"> protocols: </span></span></span><span class="line"><span class="cl"><span class="go"> forward: yes </span></span></span><span class="line"><span class="cl"><span class="go"> masquerade: yes </span></span></span><span class="line"><span class="cl"><span class="go"> forward-ports: </span></span></span><span class="line"><span class="cl"><span class="go"> port=80:proto=tcp:toport=3000:toaddr=127.0.0.1 </span></span></span><span class="line"><span class="cl"><span class="go"> source-ports: </span></span></span><span class="line"><span class="cl"><span class="go"> icmp-blocks: </span></span></span><span class="line"><span class="cl"><span class="go"> rich rules: </span></span></span></code></pre></div><p>Test a connection to port 80 with <code>curl</code> and it should redirect to the service on port 3000.</p> <p>🚨 <strong>If everything works, remember to save the firewalld configuration:</strong></p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo firewall-cmd --runtime-to-permanent </span></span><span class="line"><span class="cl"><span class="go">success </span></span></span></code></pre></div><h2 id="extra-credit" class="relative group">Extra credit <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#extra-credit" aria-label="Anchor">#</a></span></h2><p>We can inspect the nftables rules to see the firewall rules that firewalld set for us. The <a href="https://wiki.archlinux.org/title/Nftables" target="_blank" rel="noreferrer">Arch Linux nftables wiki page</a> is superb for looking up those commands.</p> <p>If we dump the current ruleset, we see the rule we created in firewalld:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo nft list ruleset </span></span><span class="line"><span class="cl"><span class="go">---SNIP--- </span></span></span><span class="line"><span class="cl"><span class="go">chain nat_PRE_FedoraServer_allow { </span></span></span><span class="line"><span class="cl"><span class="go"> meta nfproto ipv4 tcp dport 80 dnat ip to 127.0.0.1:3000 </span></span></span><span class="line"><span class="cl"><span class="go">} </span></span></span><span class="line"><span class="cl"><span class="go">---SNIP--- </span></span></span></code></pre></div>amazon-ec2-utils in Fedorahttps://major.io/p/amazon-ec2-utils-fedora/Wed, 08 May 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/amazon-ec2-utils-fedora/<p>We&rsquo;ve all been in that situation where we see a device in Linux and wonder which physical device it corresponds to. I remember when I built my first NAS and received an alert that a drive had failed. It took me a while to figure out which physical drive actually needed to be replaced.</p> <p>This happens with network devices, too, and I <a href="https://major.io/p/understanding-systemds-predictable-network-device-names/">wrote a post</a> about systemd&rsquo;s predictable network device names back in 2015.</p> <p>Cloud instances often make it even more confusing because storage devices are fully virtualized and show up differently depending on the cloud provider. I recently packaged <a href="https://github.com/amazonlinux/amazon-ec2-utils" target="_blank" rel="noreferrer">amazon-ec2-utils</a> in Fedora to make this a little easier on AWS.</p> <h2 id="the-problem" class="relative group">The problem <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#the-problem" aria-label="Anchor">#</a></span></h2><p>I just built a test instance of Fedora 40 in AWS and the AWS API shows the block device mappings like this:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> aws ec2 describe-instances <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span><span class="go"> --instance-ids i-0687448a184ab0a9e | \ </span></span></span><span class="line"><span class="cl"><span class="go"> jq &#39;.Reservations[0].Instances[0].BlockDeviceMappings&#39; </span></span></span><span class="line"><span class="cl"><span class="go">[ </span></span></span><span class="line"><span class="cl"><span class="go"> { </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;DeviceName&#34;: &#34;/dev/sda1&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;Ebs&#34;: { </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;AttachTime&#34;: &#34;2024-05-08T15:24:03+00:00&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;DeleteOnTermination&#34;: true, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;Status&#34;: &#34;attached&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;VolumeId&#34;: &#34;vol-0832569729b6c5ea6&#34; </span></span></span><span class="line"><span class="cl"><span class="go"> } </span></span></span><span class="line"><span class="cl"><span class="go"> } </span></span></span><span class="line"><span class="cl"><span class="go">] </span></span></span></code></pre></div><p>However, if I check these devices inside the instance itself, I get something totally different:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[fedora@f40 ~]$</span> sudo fdisk -l </span></span><span class="line"><span class="cl"><span class="go">Disk /dev/nvme0n1: 10 GiB, 10737418240 bytes, 20971520 sectors </span></span></span><span class="line"><span class="cl"><span class="go">Disk model: Amazon Elastic Block Store </span></span></span><span class="line"><span class="cl"><span class="go">Units: sectors of 1 * 512 = 512 bytes </span></span></span><span class="line"><span class="cl"><span class="go">Sector size (logical/physical): 512 bytes / 512 bytes </span></span></span><span class="line"><span class="cl"><span class="go">I/O size (minimum/optimal): 4096 bytes / 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go">Disklabel type: gpt </span></span></span><span class="line"><span class="cl"><span class="go">Disk identifier: 9FB58ED7-7581-4469-BEB7-64F069151EAF </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Device Start End Sectors Size Type </span></span></span><span class="line"><span class="cl"><span class="go">/dev/nvme0n1p1 2048 206847 204800 100M EFI System </span></span></span><span class="line"><span class="cl"><span class="go">/dev/nvme0n1p2 206848 2254847 2048000 1000M Linux extended boot </span></span></span><span class="line"><span class="cl"><span class="go">/dev/nvme0n1p3 2254848 20971484 18716637 8.9G Linux root (ARM-64) </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Disk /dev/zram0: 1.78 GiB, 1909456896 bytes, 466176 sectors </span></span></span><span class="line"><span class="cl"><span class="go">Units: sectors of 1 * 4096 = 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go">Sector size (logical/physical): 4096 bytes / 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go">I/O size (minimum/optimal): 4096 bytes / 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="gp">[fedora@f40 ~]$</span> ls -al /dev/sd* </span></span><span class="line"><span class="cl"><span class="go">ls: cannot access &#39;/dev/sd*&#39;: No such file or directory </span></span></span></code></pre></div><p>One disk isn&rsquo;t so bad, but what if we add more storage? The API tells me one thing:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">&gt;</span> aws ec2 describe-instances --instance-ids i-0687448a184ab0a9e <span class="p">|</span> jq <span class="s1">&#39;.Reservations[0].Instances[0].BlockDeviceMappings&#39;</span> </span></span><span class="line"><span class="cl"><span class="go">[ </span></span></span><span class="line"><span class="cl"><span class="go"> { </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;DeviceName&#34;: &#34;/dev/sda1&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;Ebs&#34;: { </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;AttachTime&#34;: &#34;2024-05-08T15:24:03+00:00&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;DeleteOnTermination&#34;: true, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;Status&#34;: &#34;attached&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;VolumeId&#34;: &#34;vol-0832569729b6c5ea6&#34; </span></span></span><span class="line"><span class="cl"><span class="go"> } </span></span></span><span class="line"><span class="cl"><span class="go"> }, </span></span></span><span class="line"><span class="cl"><span class="go"> { </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;DeviceName&#34;: &#34;/dev/sde&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;Ebs&#34;: { </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;AttachTime&#34;: &#34;2024-05-08T15:38:29.754000+00:00&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;DeleteOnTermination&#34;: false, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;Status&#34;: &#34;attached&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;VolumeId&#34;: &#34;vol-0a7ba05c5270d7aa3&#34;, </span></span></span><span class="line"><span class="cl"><span class="go"> &#34;VolumeOwnerId&#34;: &#34;xxx&#34; </span></span></span><span class="line"><span class="cl"><span class="go"> } </span></span></span><span class="line"><span class="cl"><span class="go"> } </span></span></span><span class="line"><span class="cl"><span class="go">] </span></span></span></code></pre></div><p>But then the instance tells me something else entirely:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[fedora@f40 ~]$</span> sudo fdisk -l </span></span><span class="line"><span class="cl"><span class="go">Disk /dev/nvme0n1: 10 GiB, 10737418240 bytes, 20971520 sectors </span></span></span><span class="line"><span class="cl"><span class="go">Disk model: Amazon Elastic Block Store </span></span></span><span class="line"><span class="cl"><span class="go">Units: sectors of 1 * 512 = 512 bytes </span></span></span><span class="line"><span class="cl"><span class="go">Sector size (logical/physical): 512 bytes / 512 bytes </span></span></span><span class="line"><span class="cl"><span class="go">I/O size (minimum/optimal): 4096 bytes / 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go">Disklabel type: gpt </span></span></span><span class="line"><span class="cl"><span class="go">Disk identifier: 9FB58ED7-7581-4469-BEB7-64F069151EAF </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Device Start End Sectors Size Type </span></span></span><span class="line"><span class="cl"><span class="go">/dev/nvme0n1p1 2048 206847 204800 100M EFI System </span></span></span><span class="line"><span class="cl"><span class="go">/dev/nvme0n1p2 206848 2254847 2048000 1000M Linux extended boot </span></span></span><span class="line"><span class="cl"><span class="go">/dev/nvme0n1p3 2254848 20971484 18716637 8.9G Linux root (ARM-64) </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Disk /dev/zram0: 1.78 GiB, 1909456896 bytes, 466176 sectors </span></span></span><span class="line"><span class="cl"><span class="go">Units: sectors of 1 * 4096 = 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go">Sector size (logical/physical): 4096 bytes / 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go">I/O size (minimum/optimal): 4096 bytes / 4096 bytes </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Disk /dev/nvme1n1: 10 GiB, 10737418240 bytes, 20971520 sectors </span></span></span><span class="line"><span class="cl"><span class="go">Disk model: Amazon Elastic Block Store </span></span></span><span class="line"><span class="cl"><span class="go">Units: sectors of 1 * 512 = 512 bytes </span></span></span><span class="line"><span class="cl"><span class="go">Sector size (logical/physical): 512 bytes / 512 bytes </span></span></span><span class="line"><span class="cl"><span class="go">I/O size (minimum/optimal): 4096 bytes / 4096 bytes </span></span></span></code></pre></div><h2 id="udev-rules-to-the-rescue" class="relative group">udev rules to the rescue <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#udev-rules-to-the-rescue" aria-label="Anchor">#</a></span></h2><p>The amazon-ec2-utils package provides some helpful udev rules and scripts to make it easier to identify these devices. This package is on the way to Fedora as I write this post, but it hasn&rsquo;t reached the stable repos yet. Once it does, you should be able to install it:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo dnf install amazon-ec2-utils </span></span></code></pre></div><p>In the meantime, you can download the latest build and install it on your instance:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo dnf install /usr/bin/koji </span></span><span class="line"><span class="cl"><span class="gp">$</span> koji download-build amazon-ec2-utils-2.2.0-2.fc40 </span></span><span class="line"><span class="cl"><span class="go">Downloading [1/2]: amazon-ec2-utils-2.2.0-2.fc40.src.rpm </span></span></span><span class="line"><span class="cl"><span class="go">[====================================] 100% 24.01 KiB / 24.01 KiB </span></span></span><span class="line"><span class="cl"><span class="go">Downloading [2/2]: amazon-ec2-utils-2.2.0-2.fc40.noarch.rpm </span></span></span><span class="line"><span class="cl"><span class="go">[====================================] 100% 20.53 KiB / 20.53 KiB </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="gp">$</span> sudo dnf install amazon-ec2-utils-2.2.0-2.fc40.noarch.rpm </span></span></code></pre></div><p>The cleanest method to get these new udev rules working is to reboot, but if you&rsquo;re in a hurry, there&rsquo;s an option to reload these rules without a reboot:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo udevadm control --reload-rules </span></span><span class="line"><span class="cl"><span class="gp">$</span> sudo udevadm trigger </span></span></code></pre></div><p>What do we have in <code>/dev/</code> now?</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[fedora@f40 ~]$</span> ls -al /dev/sd* </span></span><span class="line"><span class="cl"><span class="go">lrwxrwxrwx. 1 root root 7 May 8 15:44 /dev/sda1 -&gt; nvme0n1 </span></span></span><span class="line"><span class="cl"><span class="go">lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda11 -&gt; nvme0n1p1 </span></span></span><span class="line"><span class="cl"><span class="go">lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda12 -&gt; nvme0n1p2 </span></span></span><span class="line"><span class="cl"><span class="go">lrwxrwxrwx. 1 root root 9 May 8 15:44 /dev/sda13 -&gt; nvme0n1p3 </span></span></span><span class="line"><span class="cl"><span class="go">lrwxrwxrwx. 1 root root 7 May 8 15:44 /dev/sde -&gt; nvme1n1 </span></span></span></code></pre></div><p>We can put a filesystem down on the new device using the same name as the API presents:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo dnf install /usr/sbin/mkfs.btrfs </span></span><span class="line"><span class="cl"><span class="gp">$</span> sudo mkfs.btrfs /dev/sde </span></span><span class="line"><span class="cl"><span class="go">btrfs-progs v6.8.1 </span></span></span><span class="line"><span class="cl"><span class="go">See https://btrfs.readthedocs.io for more information. </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Performing full device TRIM /dev/sde (10.00GiB) ... </span></span></span><span class="line"><span class="cl"><span class="go">NOTE: several default settings have changed in version 5.15, please make sure </span></span></span><span class="line"><span class="cl"><span class="go"> this does not affect your deployments: </span></span></span><span class="line"><span class="cl"><span class="go"> - DUP for metadata (-m dup) </span></span></span><span class="line"><span class="cl"><span class="go"> - enabled no-holes (-O no-holes) </span></span></span><span class="line"><span class="cl"><span class="go"> - enabled free-space-tree (-R free-space-tree) </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Label: (null) </span></span></span><span class="line"><span class="cl"><span class="go">UUID: c2fb9e33-3bf6-4b5b-aa80-44e315f499de </span></span></span><span class="line"><span class="cl"><span class="go">Node size: 16384 </span></span></span><span class="line"><span class="cl"><span class="go">Sector size: 4096 (CPU page size: 4096) </span></span></span><span class="line"><span class="cl"><span class="go">Filesystem size: 10.00GiB </span></span></span><span class="line"><span class="cl"><span class="go">Block group profiles: </span></span></span><span class="line"><span class="cl"><span class="go"> Data: single 8.00MiB </span></span></span><span class="line"><span class="cl"><span class="go"> Metadata: DUP 256.00MiB </span></span></span><span class="line"><span class="cl"><span class="go"> System: DUP 8.00MiB </span></span></span><span class="line"><span class="cl"><span class="go">SSD detected: yes </span></span></span><span class="line"><span class="cl"><span class="go">Zoned device: no </span></span></span><span class="line"><span class="cl"><span class="go">Features: extref, skinny-metadata, no-holes, free-space-tree </span></span></span><span class="line"><span class="cl"><span class="go">Checksum: crc32c </span></span></span><span class="line"><span class="cl"><span class="go">Number of devices: 1 </span></span></span><span class="line"><span class="cl"><span class="go">Devices: </span></span></span><span class="line"><span class="cl"><span class="go"> ID SIZE PATH </span></span></span><span class="line"><span class="cl"><span class="go"> 1 10.00GiB /dev/sde </span></span></span></code></pre></div><p>Being able to know these device names during the instance launch or during storage operations makes it much easier to write automation for these devices. There&rsquo;s no guess work required to translate the device that an instance shows you to what you see via the API.</p>Fix big cursors in Java applications in Waylandhttps://major.io/p/java-big-cursors-wayland/Fri, 26 Apr 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/java-big-cursors-wayland/<p>Scroll through the list of <a href="https://major.io/tags/wayland/">Wayland posts</a> posts on the blog and you&rsquo;ll see that I&rsquo;ve solved plenty of weird problems with Wayland and the <a href="https://swaywm.org/" target="_blank" rel="noreferrer">Sway</a> compositor. Most are pretty easy to fix but some are a bit trickier.</p> <p>Java applications are notoriously unpredictable and Wayland takes unpredictability to the next level. One particular application on my desktop always seems to start with massive cursors.</p> <p>This post is about how I fixed and then discovered something interesting along the way.</p> <h2 id="fixing-big-cursors" class="relative group">Fixing big cursors <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#fixing-big-cursors" aria-label="Anchor">#</a></span></h2><p>I recently moved some investment and trading accounts from TD Ameritrade to <a href="https://tastytrade.com/" target="_blank" rel="noreferrer">Tastytrade</a>. Both offer Java applications that make trading easier, but Tastytrade&rsquo;s application always started with massive cursors.</p> <p>To make matters worse, sometimes the cursor looked lined up on the screen but then the click landed on the wrong buttons in the application! Errors are annoying. Errors that cost you money and time must be fixed. 😜</p> <p>Some web searches eventually led me to Arch Linux&rsquo;s excellent <a href="https://wiki.archlinux.org/title/Wayland" target="_blank" rel="noreferrer">Wayland wiki page</a>. None of the adjustments or environment variables there had any effect on my cursors.</p> <p>I eventually landed on a page that suggested setting <code>XCURSOR_SIZE</code>. I don&rsquo;t remember ever setting that, but it was being set by <em>something</em>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> <span class="nb">echo</span> <span class="nv">$XCURSOR_SIZE</span> </span></span><span class="line"><span class="cl"><span class="go">24 </span></span></span></code></pre></div><p>One of the suggestions was to decrease it, so I decided to give <code>20</code> a try. That was too big, but <code>16</code> was perfect and it matched all of my other applications:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> <span class="nb">export</span> <span class="nv">XCURSOR_SIZE</span><span class="o">=</span><span class="m">20</span> </span></span><span class="line"><span class="cl"><span class="gp">#</span> /opt/tastytrade/bin/tastytrade </span></span></code></pre></div><p>That works fine when I start my application via the terminal, but how do I set it for the application when I start it from ulauncher in sway? πŸ€”</p> <h2 id="desktop-file" class="relative group">Desktop file <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#desktop-file" aria-label="Anchor">#</a></span></h2><p>The Tastytade RPM comes with a <code>.desktop</code> file for launching the application. I copied that over to my local applications directory:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">cp /opt/tastytrade/lib/tastytrade-tastytrade.desktop <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> ~/.local/share/applications/ </span></span></code></pre></div><p>Then I opened the copied <code>~/.local/share/applications/tastytrade-tastytrade.desktop</code> file in a text editor:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Desktop Entry]</span> </span></span><span class="line"><span class="cl"><span class="na">Name</span><span class="o">=</span><span class="s">tastytrade</span> </span></span><span class="line"><span class="cl"><span class="na">Comment</span><span class="o">=</span><span class="s">tastytrade</span> </span></span><span class="line"><span class="cl"><span class="na">Exec</span><span class="o">=</span><span class="s">/opt/tastytrade/bin/tastytrade</span> </span></span><span class="line"><span class="cl"><span class="na">Icon</span><span class="o">=</span><span class="s">/opt/tastytrade/lib/tastytrade.png</span> </span></span><span class="line"><span class="cl"><span class="na">Terminal</span><span class="o">=</span><span class="s">false</span> </span></span><span class="line"><span class="cl"><span class="na">Type</span><span class="o">=</span><span class="s">Application</span> </span></span><span class="line"><span class="cl"><span class="na">Categories</span><span class="o">=</span><span class="s">tastyworks</span> </span></span><span class="line"><span class="cl"><span class="na">MimeType</span><span class="o">=</span> </span></span></code></pre></div><p>I changed the <code>Exec</code> line to be:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="na">Exec</span><span class="o">=</span><span class="s">env XCURSOR_SIZE=16 /opt/tastytrade/bin/tastytrade</span> </span></span></code></pre></div><p>I launched the application again after making that change, but the cursors were still huge! There has to be another way. πŸ€”</p> <h2 id="systemd-does-everything-" class="relative group">systemd does everything πŸ˜† <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#systemd-does-everything-" aria-label="Anchor">#</a></span></h2><p>After more searching and digging, I discovered that systemd has a capability to <a href="https://www.freedesktop.org/software/systemd/man/latest/environment.d.html" target="_blank" rel="noreferrer">set environment variables for user sessions</a>:</p> <blockquote> <p>Configuration files in the environment.d/ directories contain lists of environment variable assignments passed to services started by the systemd user instance. systemd-environment-d-generator(8) parses them and updates the environment exported by the systemd user instance. See below for an discussion of which processes inherit those variables.</p> <p>It is recommended to use numerical prefixes for file names to simplify ordering.</p> <p>For backwards compatibility, a symlink to /etc/environment is installed, so this file is also parsed.</p> </blockquote> <p>Let&rsquo;s give that a try:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> mkdir -p ~/.config/environment.d/ </span></span><span class="line"><span class="cl"><span class="gp">$</span> vim ~/.config/environment.d/wayland.conf </span></span></code></pre></div><p>In the file, I added one line with a comment (because you will soon forget why you added it πŸ˜„):</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="c1"># Fix big cursors in Java apps in Wayland</span> </span></span><span class="line"><span class="cl"><span class="nv">XCURSOR_SIZE</span><span class="o">=</span><span class="m">16</span> </span></span></code></pre></div><p><strong>After a reboot, I launched my Java application and boom &ndash; the cursors were perfect!</strong> πŸŽ‰</p> <p>I went back and cleaned up some other hacks I had applied and added them to that <code>wayland.conf</code> file:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="c1"># This was important at some point but I&#39;m afraid to remove it.</span> </span></span><span class="line"><span class="cl"><span class="c1"># Note to self: make detailed comments when adding lines here.</span> </span></span><span class="line"><span class="cl"><span class="nv">SDL_VIDEODRIVER</span><span class="o">=</span>wayland </span></span><span class="line"><span class="cl"><span class="nv">QT_QPA_PLATFORM</span><span class="o">=</span>wayland </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Reduce window decorations for VLC</span> </span></span><span class="line"><span class="cl"><span class="nv">QT_WAYLAND_DISABLE_WINDOWDECORATION</span><span class="o">=</span><span class="s2">&#34;1&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Fix weird window handling when Java apps do certain pop-ups</span> </span></span><span class="line"><span class="cl"><span class="nv">_JAVA_AWT_WM_NONREPARENTING</span><span class="o">=</span><span class="m">1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Ensure Firefox is using Wayland code (not needed any more)</span> </span></span><span class="line"><span class="cl"><span class="nv">MOZ_ENABLE_WAYLAND</span><span class="o">=</span><span class="m">1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Disable HiDPI</span> </span></span><span class="line"><span class="cl"><span class="nv">GDK_SCALE</span><span class="o">=</span><span class="m">1</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Fix big cursors in Java apps in Wayland</span> </span></span><span class="line"><span class="cl"><span class="nv">XCURSOR_SIZE</span><span class="o">=</span><span class="m">16</span> </span></span></code></pre></div><p>I&rsquo;m told there are some caveats with this solution, especially if your Wayland desktop doesn&rsquo;t use systemd to start. This is working for me with GDM launching Sway on Fedora 40.</p>cloud-init and dhcpcdhttps://major.io/p/fedora-cloud-init-dhcpcd/Thu, 18 Apr 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/fedora-cloud-init-dhcpcd/<p>We&rsquo;re all familiar with the trusty old <code>dhclient</code> on our Linux systems, but <a href="https://github.com/isc-projects/dhcp" target="_blank" rel="noreferrer">it went end-of-life in 2022</a>:</p> <pre tabindex="0"><code>NOTE: This software is now End-Of-Life. 4.4.3 is the final release planned. We will continue to keep the public issue tracker and user mailing list open. You should read this file carefully before trying to install or use the ISC DHCP Distribution. </code></pre><p>Most Linux distributions use <code>dhclient</code> along with cloud-init for the initial dhcp request during the first part of cloud-init&rsquo;s work. I set off to switch Fedora&rsquo;s cloud-init package to <code>dhcpcd</code> instead.</p> <h2 id="whats-new-with-dhcpcd" class="relative group">What&rsquo;s new with dhcpcd? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#whats-new-with-dhcpcd" aria-label="Anchor">#</a></span></h2><p>There are some nice things about <code>dhcpcd</code> that you can find in the <a href="https://github.com/NetworkConfiguration/dhcpcd" target="_blank" rel="noreferrer">GitHub repository</a>:</p> <ul> <li>Very small footprint with almost no dependencies on Fedora</li> <li>It can do DHCP and DHCPv6</li> <li>It can also be a <a href="https://en.wikipedia.org/wiki/Zeroconf" target="_blank" rel="noreferrer">ZeroConf</a> client</li> </ul> <p>The project had its last release back in December 2023 and had commits as recently as this week.</p> <h2 id="but-i-use-networkmanager" class="relative group">But I use NetworkManager <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#but-i-use-networkmanager" aria-label="Anchor">#</a></span></h2><p>That&rsquo;s great! A switch from <code>dhclient</code> to <code>dhcpcd</code> for cloud-init won&rsquo;t affect you.</p> <p>When cloud-init starts, it does an initial dhcp request to get just enough networking to reach the cloud&rsquo;s metadata service. This service provides all kinds of information for cloud-init, including network setup instructions and initial scripts to run.</p> <p>NetworkManager doesn&rsquo;t start taking action until cloud-init has written the network configuration to the system.</p> <h2 id="but-i-use-systemd-networkd" class="relative group">But I use systemd-networkd <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#but-i-use-systemd-networkd" aria-label="Anchor">#</a></span></h2><p>Same as with NetworkManager, this change applies to the <em>very</em> early boot and you won&rsquo;t notice a different when deploying new cloud systems.</p> <h2 id="how-can-i-get-it-right-now" class="relative group">How can I get it right now? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#how-can-i-get-it-right-now" aria-label="Anchor">#</a></span></h2><p>If you&rsquo;re using a recent build of Fedora rawhide (the unstable release under development), you likely have it right now on your cloud instance. Just run <code>journalctl --boot</code>, search for <code>dhcpcd</code>, and you should see these lines:</p> <pre tabindex="0"><code>cloud-init[725]: Cloud-init v. 24.1.4 running &#39;init-local&#39; at Wed, 17 Apr 2024 14:39:36 +0000. Up 6.13 seconds. dhcpcd[727]: dhcpcd-10.0.6 starting kernel: 8021q: 802.1Q VLAN Support v1.8 dhcpcd[730]: DUID 00:01:00:01:2d:b2:9b:a9:06:eb:18:e7:22:dd dhcpcd[730]: eth0: IAID 18:e7:22:dd dhcpcd[730]: eth0: soliciting a DHCP lease dhcpcd[730]: eth0: offered 172.31.26.195 from 172.31.16.1 dhcpcd[730]: eth0: leased 172.31.26.195 for 3600 seconds avahi-daemon[706]: Joining mDNS multicast group on interface eth0.IPv4 with address 172.31.26.195. avahi-daemon[706]: New relevant interface eth0.IPv4 for mDNS. avahi-daemon[706]: Registering new address record for 172.31.26.195 on eth0.IPv4. dhcpcd[730]: eth0: adding route to 172.31.16.0/20 dhcpcd[730]: eth0: adding default route via 172.31.16.1 dhcpcd[730]: control command: /usr/sbin/dhcpcd --dumplease --ipv4only eth0 </code></pre><p>There&rsquo;s also an <a href="https://bodhi.fedoraproject.org/updates/FEDORA-2024-51d7f6b005" target="_blank" rel="noreferrer">update pending for Fedora 40</a>, but it&rsquo;s currently held up by the beta freeze. That should appear as an update as soon as Fedora 40 is released.</p> <p>Keep in mind that if you have a system deployed already, cloud-init won&rsquo;t need to run again. Updating to Fedora 40 will update your cloud-init and pull in <code>dhcpcd</code>, but it won&rsquo;t need to run again since your configuration is already set.</p>Texas Linux Fest 2024 recap 🀠https://major.io/p/texas-linux-fest-2024-recap/Tue, 16 Apr 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/texas-linux-fest-2024-recap/<p>The 2024 <a href="https://2024.texaslinuxfest.org/" target="_blank" rel="noreferrer">Texas Linux Festival</a> just ended last weekend and it was a fun event as always. It&rsquo;s one my favorite events to attend because it&rsquo;s really casual. You have plenty of opportunities to see old friends, meet new people, and learn a few things along the way.</p> <p>I was fortunate enough to have two talks accepted for this year&rsquo;s event. One was focused on containers while the other was a (very belated) addition to my <a href="https://major.io/p/impostor-syndrome-talk-faqs-and-follow-ups/">impostor syndrome talk</a> from 2015.</p> <p>This was also my first time building slides with <a href="https://github.com/webpro/reveal-md" target="_blank" rel="noreferrer">reveal-md</a>, a &ldquo;batteries included&rdquo; package for making <a href="https://revealjs.com/" target="_blank" rel="noreferrer">reveal.js</a> slides. Nothing broke too badly and that was a relief.</p> <h2 id="containers-talk" class="relative group">Containers talk <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#containers-talk" aria-label="Anchor">#</a></span></h2><p>I&rsquo;ve wanted to share more of what I&rsquo;ve done with CoreOS in low-budget container deployments and this seemed like a good time to share it with the world out loud. My talk, <a href="https://txlf24-containers.major.io/#/" target="_blank" rel="noreferrer">Automated container updates with GitHub and CoreOS</a>, walked the audience through how to deploy containers on CoreOS, keep them updated, and update the container image source.</p> <p>My goal was to keep it as low on budget as possible. Much of it was centered around a stack of <a href="https://major.io/p/caddy-porkbun/">caddy</a>, librespeed, and docker-compose. All of it was kept up to date with <a href="https://major.io/p/watchtower/">watchtower</a>.</p> <p>My custom Caddy container needed support for <a href="https://porkbun.com/" target="_blank" rel="noreferrer">Porkbun&rsquo;s</a> DNS API and I used GitHub Actions to build that container and serve it to the internet using GitHub&rsquo;s package hosting. <em>This also gave me the opportunity to share how awesome Porkbun is for registering domains, including their <a href="https://porkbun.com/tld/jobs" target="_blank" rel="noreferrer">customized pig artwork</a> for every TLD imaginable.</em> 🐷</p> <p>We had a great discussion afterwards about how CoreOS <strong>does indeed live on</strong> as <a href="https://fedoraproject.org/coreos/" target="_blank" rel="noreferrer">Fedora CoreOS</a>.</p> <h2 id="tech-career-talk" class="relative group">Tech career talk <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#tech-career-talk" aria-label="Anchor">#</a></span></h2><p>This talk made me nervous because it had a lot of slides to cover, but I also wanted to leave plenty of time for questions. <a href="https://txlf24-tech-career.major.io/#/" target="_blank" rel="noreferrer">Five tips for a thriving technology career</a> built upon my old impostor syndrome talk by sharing some of the things I&rsquo;ve learned over the year that helped me succeed in my career.</p> <p>I managed to end early with time for questions, and boy did the audience have questions! πŸ“£ Some audience members helped me answer some questions, too!</p> <p>We talked a lot about office politics, tribal knowledge, and toxic workplaces. The audience generally agreed that most businesses tried to rub copious amounts of Confluence on their tribal knowledge problem, but it never improved. 😜</p> <p>The room was full with people standing in the back and I&rsquo;m tremendously humbled by everyone who came. I received plenty of feedback afterwards and that&rsquo;s the best gift I could ever get. 🎁</p> <h2 id="other-talks" class="relative group">Other talks <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#other-talks" aria-label="Anchor">#</a></span></h2><p><a href="https://github.com/anitazha" target="_blank" rel="noreferrer">Anita Zhang</a> had an excellent keynote talk on the second day about her unusual path into the world of technology. Her slides were pictures of her dog that lined up with various parts of her story. That was a great idea.</p> <p><a href="https://www.linkedin.com/in/kyle-davis-linux/?originalSubdomain=ca" target="_blank" rel="noreferrer">Kyle Davis</a> offered talks on <a href="https://github.com/valkey-io/valkey" target="_blank" rel="noreferrer">valkey</a> and <a href="https://github.com/bottlerocket-os/bottlerocket" target="_blank" rel="noreferrer">bottlerocket</a>. There was plenty about the redis and valkey story that I didn&rsquo;t know and the context was useful. It looks like you can simply drop valkey into most redis environments without much disruption.</p> <p><a href="https://www.linkedin.com/in/thomascameron/" target="_blank" rel="noreferrer">Thomas Cameron</a> talked about running OKD on Fedora CoreOS in his home lab. There were quite a few steps, but he did a great job of connecting the dots between what needed to be done and why.</p> <h2 id="around-the-exhibit-hall" class="relative group">Around the exhibit hall <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#around-the-exhibit-hall" aria-label="Anchor">#</a></span></h2><p>I helped staff the Fedora/CoreOS booth and we had plenty of questions. Most questions were around the M1 Macbook running <a href="https://asahilinux.org/" target="_blank" rel="noreferrer">Asahi Linux</a> that was on the table. πŸ˜‰</p> <p>There were still quite a few misconceptions around the CentOS Stream changes, as well as how AlmaLinux and Rocky Linux fit into the picture. Our booth was right next to the AlmaLinux booth and I had the opportunity to meet <a href="https://jonathanspw.com/about/" target="_blank" rel="noreferrer">Jonathan Wright</a>. That was awesome!</p> <p><strong>I can&rsquo;t wait for next year&rsquo;s event.</strong></p>Roll your own static blog analyticshttps://major.io/p/static-blog-analytics/Thu, 04 Apr 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/static-blog-analytics/<p>Static blogs come with tons of advantages. They&rsquo;re cheap to serve. You store all your changes in git. People with spotty internet connections can clone your blog and run it locally.</p> <p><strong>However, one of the challenges that I&rsquo;ve run into over the years is around analytics.</strong></p> <p>I could quickly add Google Analytics to the site and call it a day, but is that a good idea? Many browsers have ad blocking these days and the analytics wouldn&rsquo;t even run. For those that don&rsquo;t have an ad blocker, do I want to send more data about them to Google? πŸ™ƒ</p> <p>How about running my own self-hosted analytics platform? That&rsquo;s pretty easy with containers, but most ad blockers know about those, too.</p> <p>This post talks about how to host a static blog in a container behind a <a href="https://caddyserver.com/" target="_blank" rel="noreferrer">Caddy</a> web server. We will use <a href="https://goaccess.io/" target="_blank" rel="noreferrer">goaccess</a> to analyze the log files on the server itself to avoid dragging in an analytics platform.</p> <h2 id="why-do-you-need-analytics" class="relative group">Why do you need analytics? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#why-do-you-need-analytics" aria-label="Anchor">#</a></span></h2><p>Yes, yes, I know this comes from the guy who wrote a post about <a href="https://major.io/p/how-i-write-blog-posts/">writing for yourself</a>, but sometimes I like to know which posts are popular with other people. I also like to know if something&rsquo;s misconfigured and visitors are seeing 404 errors for pages which should be working.</p> <p>It can also be handy to know when someone else is <a href="https://major.io/p/puppy-linux-icanhazip-and-tin-foil-hats/">writing about you</a>, especially when those things are incorrect. πŸ˜‰</p> <p>So my goals here are these:</p> <ul> <li>Get some basic data on what&rsquo;s resonating with people and what isn&rsquo;t</li> <li>Find configuration errors that are leading visitors to error pages</li> <li>Learn more about who is linking to the site</li> <li>Do all this without impacting user privacy through heavy javascript trackers</li> </ul> <h2 id="what-are-the-ingredients" class="relative group">What are the ingredients? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#what-are-the-ingredients" aria-label="Anchor">#</a></span></h2><p>There are three main pieces:</p> <ol> <li>Caddy, a small web server that runs really well in containers</li> <li>This blog, which is written with <a href="https://gohugo.io/" target="_blank" rel="noreferrer">Hugo</a> and <a href="https://github.com/major/major.io" target="_blank" rel="noreferrer">stored in GitHub</a></li> <li>Goaccess, a log analyzer with a capability to do live updates via websockets</li> </ol> <p>Caddy will write logs to a location that goaccess can read. In turn, goaccess will write log analysis to an HTML file that caddy can serve. The HTML file served by caddy will open a websocket to goaccess for live analytics.</p> <h2 id="a-static-blog-in-a-container" class="relative group">A static blog in a container? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#a-static-blog-in-a-container" aria-label="Anchor">#</a></span></h2><p>We can pack a static blog into a very thin container with an extremely lightweight web server. After all, caddy can handle automatic TLS certificate installation, logging, and caching. That just means we need the most basic webserver in the container itself.</p> <p>I was considering a second caddy container with the blog content in it until I stumbled upon a great post by Florin Lipan about <a href="https://lipanski.com/posts/smallest-docker-image-static-website" target="_blank" rel="noreferrer">The smallest Docker image to serve static websites</a>. He went down a rabbit hole to make the smallest possible web server container with busybox.</p> <p>His first stop led to a 1.25MB container, and that&rsquo;s tiny enough for me.<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> 🀏</p> <p>I built a <a href="https://github.com/major/major.io/blob/main/.github/workflows/container.yml" target="_blank" rel="noreferrer">container workflow</a> in GitHub Actions that builds a container, puts the blog in it, and <a href="https://github.com/major/major.io/pkgs/container/major.io" target="_blank" rel="noreferrer">stores that container as a package</a> in the GitHub repository. It all starts with a brief Dockerfile:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-Dockerfile" data-lang="Dockerfile"><span class="line"><span class="cl"><span class="k">FROM</span><span class="s"> docker.io/library/busybox:1.36.1</span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">RUN</span> adduser -D static<span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">USER</span><span class="s"> static</span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">WORKDIR</span><span class="s"> /home/static</span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">COPY</span> ./public/ /home/static<span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">CMD</span> <span class="p">[</span><span class="s2">&#34;busybox&#34;</span><span class="p">,</span> <span class="s2">&#34;httpd&#34;</span><span class="p">,</span> <span class="s2">&#34;-f&#34;</span><span class="p">,</span> <span class="s2">&#34;-p&#34;</span><span class="p">,</span> <span class="s2">&#34;3000&#34;</span><span class="p">]</span><span class="err"> </span></span></span></code></pre></div><p>We start with busybox, add a user, put the website content into the user&rsquo;s home directory, and start busybox&rsquo;s <code>httpd</code> server. The container starts up and serves the static content on port 3000.</p> <h2 id="caddy-logs" class="relative group">Caddy logs <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#caddy-logs" aria-label="Anchor">#</a></span></h2><p>Caddy writes its logs in a JSON format and goaccess already knows how to parse caddy logs. Our first step is to get caddy writing some logs. In my case, I have a directory called <code>caddy/logs/</code> in my home directory where those logs are written.</p> <p>I&rsquo;ll mount the log storage into the caddy container and mount one extra directory to hold the HTML file that goaccess will write. Here&rsquo;s my <code>docker-compose.yaml</code> excerpt:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w"> </span><span class="nt">caddy</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">ghcr.io/major/caddy:main</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">caddy</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">&#34;80:80/tcp&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">&#34;443:443/tcp&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">&#34;443:443/udp&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">unless-stopped</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">./caddy/Caddyfile:/etc/caddy/Caddyfile:Z</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">caddy_data:/data</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">caddy_config:/config</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Caddy writes logs here πŸ‘‡</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">./caddy/logs:/logs:z</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># This is for goaccess to write its HTML file πŸ‘‡</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">./storage/goaccess_major_io:/var/www/goaccess_major_io:z</span><span class="w"> </span></span></span></code></pre></div><p>Now we need to update the <code>Caddyfile</code> to tell caddy where to place the logs and add a <code>reverse_proxy</code> configuration for our new container that serves the blog:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-Caddyfile" data-lang="Caddyfile"><span class="line"><span class="cl"><span class="gh">major.io</span> <span class="p">{</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> # We will set up this container in a moment πŸ‘‡ </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">reverse_proxy</span> <span class="s">major_io:3000</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">lb_try_duration</span> <span class="mi">30s</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> # Tell Caddy to write logs to `/logs` which </span></span></span><span class="line"><span class="cl"><span class="c1"> # is `storage/logs` on the host: </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">log</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">output</span> <span class="s">file</span> <span class="s">/logs/major.io-access.log</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">roll_size</span> <span class="s">1024mb</span> </span></span><span class="line"><span class="cl"> <span class="k">roll_keep</span> <span class="mi">20</span> </span></span><span class="line"><span class="cl"> <span class="k">roll_keep_for</span> <span class="mi">720h</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span></code></pre></div><p>Great! We now have the configuration in place for caddy to write the logs and the caddy container can mount the log and analytics storage.</p> <h2 id="enabling-analytics" class="relative group">Enabling analytics <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#enabling-analytics" aria-label="Anchor">#</a></span></h2><p>We&rsquo;re heading back to the <code>docker-compose.yml</code> file once more, this time to set up a goaccess container:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w"> </span><span class="nt">goaccess_major_io</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/allinurl/goaccess:latest</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">goaccess_major_io</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">always</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Mount caddy&#39;s log files πŸ‘‡</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">&#34;./caddy/logs:/var/log/caddy:z&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Mount the directory where goaccess writes the analytics HTML πŸ‘‡</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="s2">&#34;./storage/goaccess_major_io:/var/www/goaccess:rw&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;/var/log/caddy/major.io-access.log --log-format=CADDY -o /var/www/goaccess/index.html --real-time-html --ws-url=wss://stats.major.io:443/ws --port=7890 --anonymize-ip --ignore-crawlers --real-os&#34;</span><span class="w"> </span></span></span></code></pre></div><p>This gets us a goaccess container to parse the logs from caddy. We need to update the caddy configuration so that we can reach the goaccess websocket for live updates:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-Caddyfile" data-lang="Caddyfile"><span class="line"><span class="cl"><span class="gh">stats.major.io</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">root</span> <span class="nd">*</span> <span class="s">/var/www/goaccess_major_io</span> </span></span><span class="line"><span class="cl"> <span class="k">file_server</span> </span></span><span class="line"><span class="cl"> <span class="k">reverse_proxy</span> <span class="nd">/ws</span> <span class="s">goaccess_major_io:7890</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span></code></pre></div><p>At this point, we have caddy writing logs in the right place, goaccess can read them, and the analytics output is written to a place where caddy can serve it. We&rsquo;ve also exposed the websocket from goaccess for live updates.</p> <h2 id="serving-the-blog" class="relative group">Serving the blog <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#serving-the-blog" aria-label="Anchor">#</a></span></h2><p>We&rsquo;ve reached the most important part!</p> <p>We added the caddy configuration to reach the blog container earlier, but now it&rsquo;s time to deploy the container itself. As a reminder, this is the container with busybox and the blog content that comes from GitHub Actions.</p> <p>The <code>docker-compose.yml</code> configuration here is <em>very basic</em>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w"> </span><span class="nt">major_io</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">ghcr.io/major/major.io:main</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">major_io</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">always</span><span class="w"> </span></span></span></code></pre></div><p>Caddy will connect to this container on port 3000 to serve the blog. (We set port 3000 in the original <code>Dockerfile</code>).</p> <p>At this point, everything should be set to go. Make it live with:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">docker-compose up -d </span></span></span></code></pre></div><p>This should bring up the goaccess and blog containers while also restarting caddy. The website should be visible now at <a href="https://major.io/" target="_blank" rel="noreferrer">major.io</a> (and that&rsquo;s how you&rsquo;re reading this today).</p> <h2 id="what-about-new-posts" class="relative group">What about new posts? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#what-about-new-posts" aria-label="Anchor">#</a></span></h2><p>I&rsquo;m glad you asked! That was something I wondered about as well. <strong>How do we get the new blog content down to the container when a new post is written?</strong> πŸ€”</p> <p>As I&rsquo;ve <a href="https://major.io/p/watchtower/">written in the past</a>, I like using <a href="https://containrrr.dev/watchtower/" target="_blank" rel="noreferrer">watchtower</a> to keep containers updated. Watchtower offers an HTTP API interface for webhooks to initiate container updates. We can trigger that update via a simple curl request from GitHub Actions when our container pipeline runs.</p> <p>My <a href="https://github.com/major/major.io/blob/main/.github/workflows/container.yml" target="_blank" rel="noreferrer">container workflow</a> has a brief bit at the end that does this:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">Update the blog container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">if</span><span class="p">:</span><span class="w"> </span><span class="l">github.event_name != &#39;pull_request&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">run</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> curl -s -H &#34;Authorization: Bearer ${WATCHTOWER_TOKEN}&#34; \ </span></span></span><span class="line"><span class="cl"><span class="sd"> https://watchtower.thetanerd.com/v1/update</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">env</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">WATCHTOWER_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${{ secrets.WATCHTOWER_TOKEN }}</span><span class="w"> </span></span></span></code></pre></div><p>You can enable this in watchtower with a few new environment variables in your <code>docker-compose.yml</code>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-YAML" data-lang="YAML"><span class="line"><span class="cl"><span class="w"> </span><span class="nt">watchtower</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># New environment variables πŸ‘‡</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">WATCHTOWER_HTTP_API_UPDATE=true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">WATCHTOWER_HTTP_API_TOKEN=SUPER-SECRET-TOKEN-PASSWORD</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">WATCHTOWER_HTTP_API_PERIODIC_POLLS=true</span><span class="w"> </span></span></span></code></pre></div><p><code>WATCHTOWER_HTTP_API_UPDATE</code> enables the updating via API and <code>WATCHTOWER_HTTP_API_TOKEN</code> sets the token required when making the API request. If you set <code>WATCHTOWER_HTTP_API_PERIODIC_POLLS</code> to <code>true</code>, watchtower will still periodically look for updates to containers even if an API request never appeared. By default, watchtower will stop doing periodic updates if you enable the API.</p> <p>This is working on my site right now and you can view my public blog stats on <a href="https://stats.major.io" target="_blank" rel="noreferrer">stats.major.io</a>. πŸŽ‰</p> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>Florin went all the way down to 154KB and I was extremely impressed. However, I&rsquo;m not too worried about an extra megabyte here. πŸ˜‰&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>Connect Caddy to Porkbunhttps://major.io/p/caddy-porkbun/Thu, 29 Feb 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/caddy-porkbun/<p>I recently told a coworker about <a href="https://caddyserver.com/" target="_blank" rel="noreferrer">Caddy</a>, a small web and proxy server with a very simple configuration. It also has a handy feature where it manages your TLS certificate for you automatically.</p> <p>However, one problem I had at home with my <a href="https://fedoraproject.org/coreos/" target="_blank" rel="noreferrer">CoreOS</a> deployment is that I don&rsquo;t have inbound network access to handle the certificate verification process. Most automated certificate vendors need to reach your web server to verify that you have control over your domain.</p> <p>This post talks about how to work around this problem with domains registered at <a href="https://porkbun.com/" target="_blank" rel="noreferrer">Porkbun</a>.</p> <h2 id="dns-validation" class="relative group">DNS validation <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#dns-validation" aria-label="Anchor">#</a></span></h2><p>Certificate providers usually default to verifying domains by making a request to your server and retrieving a validation code. If your systems are all behind a firewall without inbound access from the internet, you can use DNS validation instead.</p> <p>The process looks something like this:</p> <ol> <li>You tell the certificate provider the domain names you want on your certificate</li> <li>The certificate provider gives you some DNS records to add wherever you host your DNS records</li> <li>You add the DNS records</li> <li>You get your certificates once the certificate provider verifies the records.</li> </ol> <p>You can do this manually with something like <a href="https://github.com/acmesh-official/acme.sh" target="_blank" rel="noreferrer">acme.sh</a> today, but it&rsquo;s <strong>painful</strong>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="c1"># Make the initial certificate request</span> </span></span><span class="line"><span class="cl">acme.sh --issue --dns -d example.com <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --yes-I-know-dns-manual-mode-enough-go-ahead-please </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Add your DNS records manually.</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Verify the DNS records and issue the certificates.</span> </span></span><span class="line"><span class="cl">acme.sh --issue --dns -d example.com <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --yes-I-know-dns-manual-mode-enough-go-ahead-please --renew </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Copy the keys/certificates and configure your webserver.</span> </span></span></code></pre></div><p>We don&rsquo;t want to live this way.</p> <p> <figure> <picture class="mx-auto my-0 rounded-md" > <img width="581" height="355" class="mx-auto my-0 rounded-md" alt="do-not-want.gif" loading="lazy" decoding="async" src="https://major.io/p/caddy-porkbun/do-not-want.gif" /> </picture> </figure> </p> <p>Let&rsquo;s talk about how Caddy can help.</p> <h2 id="adding-porkbun-support-to-caddy" class="relative group">Adding Porkbun support to Caddy <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#adding-porkbun-support-to-caddy" aria-label="Anchor">#</a></span></h2><p>Caddy is a minimal webserver and <a href="https://github.com/caddy-dns/porkbun" target="_blank" rel="noreferrer">Porkbun support</a> doesn&rsquo;t get included by default. However, we can quickly add it via a simple container build:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-Dockerfile" data-lang="Dockerfile"><span class="line"><span class="cl"><span class="k">FROM</span><span class="s"> caddy:2.7.6-builder AS builder</span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">RUN</span> xcaddy build <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --with github.com/caddy-dns/porkbun<span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">FROM</span><span class="s"> caddy:2.7.6</span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="k">COPY</span> --from<span class="o">=</span>builder /usr/bin/caddy /usr/bin/caddy<span class="err"> </span></span></span></code></pre></div><p>This is a two stage container build where we compile the Porkbun support and then use that new <code>caddy</code> binary in the final container.</p> <p>We&rsquo;re not done yet!</p> <h2 id="automated-caddy-builds-with-updates" class="relative group">Automated Caddy builds with updates <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#automated-caddy-builds-with-updates" aria-label="Anchor">#</a></span></h2><p>I created a <a href="https://github.com/major/caddy" target="_blank" rel="noreferrer">GitHub repository</a> that builds the Caddy container for me and keeps it updated. There&rsquo;s a <a href="https://github.com/major/caddy/blob/main/.github/workflows/docker-publish.yml" target="_blank" rel="noreferrer">workflow to publish a container</a> to GitHub&rsquo;s container repository and I can pull containers from there on my various CoreOS machines.</p> <p>In addition, I use <a href="https://github.com/apps/renovate" target="_blank" rel="noreferrer">Renovate</a> to watch for Caddy updates. New updates come through a <a href="https://github.com/major/caddy/pull/10" target="_blank" rel="noreferrer">regular pull request</a> and I can apply them whenever I want.</p> <p> <figure> <picture class="mx-auto my-0 rounded-md" > <img width="918" height="461" class="mx-auto my-0 rounded-md" alt="Renovate pull request" loading="lazy" decoding="async" src="https://major.io/p/caddy-porkbun/pr_hu867250beddb270420f4f59a773767efc_82390_660x0_resize_box_3.png" srcset="https://major.io/p/caddy-porkbun/pr_hu867250beddb270420f4f59a773767efc_82390_330x0_resize_box_3.png 330w,/p/caddy-porkbun/pr_hu867250beddb270420f4f59a773767efc_82390_660x0_resize_box_3.png 660w ,/p/caddy-porkbun/pr.png 918w ,/p/caddy-porkbun/pr.png 918w " sizes="100vw" /> </picture> <figcaption class="text-center">Example pull request from Renovate</figcaption> </figure> </p> <h2 id="connecting-to-porkbun" class="relative group">Connecting to Porkbun <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#connecting-to-porkbun" aria-label="Anchor">#</a></span></h2><p>We start here by getting an API key to manage the domain at Porkbun.</p> <ol> <li>Log into your <a href="https://porkbun.com/account/domainsSpeedy" target="_blank" rel="noreferrer">Porkbun dashboard</a>.</li> <li>Click <strong>Details</strong> to the right of the domain you want to manage.</li> <li>Look for <strong>API Access</strong> in the leftmost column and turn it on.</li> <li>At the top right of the dashboard, click <strong>Account</strong> and then <strong>API Access</strong>.</li> <li>Add a title for your new API key, such as <em>Caddy</em>, and click <strong>Create API Key</strong>.</li> <li>Save the API key and secrey key that are displayed.</li> </ol> <p>Open up your Caddy configuration file (the <em>Caddyfile</em>) and add some configuration:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-caddyfile" data-lang="caddyfile"><span class="line"><span class="cl"><span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">email</span> <span class="s">me@example.com</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> # Uncomment this next line if you want to get </span></span></span><span class="line"><span class="cl"><span class="c1"> # some test certificates first. </span></span></span><span class="line"><span class="cl"><span class="c1"> # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory </span></span></span><span class="line"><span class="cl"><span class="c1"></span> </span></span><span class="line"><span class="cl"> <span class="k">acme_dns</span> <span class="s">porkbun</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">api_key</span> <span class="s">pk1_******</span> </span></span><span class="line"><span class="cl"> <span class="k">api_secret_key</span> <span class="s">sk1_******</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="gh">example.com</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">handle</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">respond</span> <span class="s2">&#34;Hello world!&#34;</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span></code></pre></div><p>Save the Caddyfile and restart your Caddy server or container. Caddy will immediately begin requesting your TLS certificates and managing your DNS records for those certificates. This normally finishes in less than 30 seconds or so during the first run.</p> <p>If you don&rsquo;t see the HTTPS endpoint working within a minute or two, be sure to check the Caddy logs. You might have a typo in a Porkbun API key or the domain you&rsquo;re trying to modify doesn&rsquo;t have the <strong>API Access</strong> switch enabled.</p> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 0C114.6 0 0 114.6 0 256s114.6 256 256 256s256-114.6 256-256S397.4 0 256 0zM256 128c17.67 0 32 14.33 32 32c0 17.67-14.33 32-32 32S224 177.7 224 160C224 142.3 238.3 128 256 128zM296 384h-80C202.8 384 192 373.3 192 360s10.75-24 24-24h16v-64H224c-13.25 0-24-10.75-24-24S210.8 224 224 224h32c13.25 0 24 10.75 24 24v88h16c13.25 0 24 10.75 24 24S309.3 384 296 384z"/></svg> </span> </span> <span class="dark:text-neutral-300">Remember that Porkbun requires you to enable API access for each domain. API access is disabled at Porkbun by default.</span> </div> <p><strong>That&rsquo;s it!</strong> πŸŽ‰</p> <h2 id="renewals" class="relative group">Renewals <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#renewals" aria-label="Anchor">#</a></span></h2><p>Caddy will keep watch over the certificates and begin the renewal process as the expiration approaches. It has a very careful retry mechanism that ensures your certificates are updated without tripping any rate limits at the certificate provider.</p> <h2 id="further-reading" class="relative group">Further reading <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#further-reading" aria-label="Anchor">#</a></span></h2><p>Caddy&rsquo;s detailed documentation about <a href="https://caddyserver.com/docs/automatic-https" target="_blank" rel="noreferrer">Automatic HTTPS</a> and the <a href="https://caddyserver.com/docs/caddyfile/directives/tls" target="_blank" rel="noreferrer">tls configuration directive</a> should answer most questions about how the process works.</p>Linux on the AMD ThinkPad Z13 G2https://major.io/p/linux-thinkpad-z13-amd/Sun, 14 Jan 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/linux-thinkpad-z13-amd/<p>AMD&rsquo;s new <a href="https://en.wikipedia.org/wiki/Zen_4" target="_blank" rel="noreferrer">Zen 4 processors</a> started rolling out in 2022 and I&rsquo;ve been watching for the mobile CPUs to reach laptops. I like where AMD is going with these chips and how they provide lots of CPU power without eating up the battery.</p> <p>I recently ordered a <a href="https://www.lenovo.com/us/en/p/laptops/thinkpad/thinkpadz/thinkpad-z13-gen-2-%2813-inch-amd%29/len101t0073" target="_blank" rel="noreferrer">ThinkPad Z13 Gen 2</a> with an AMD Ryzen 7. As you might expect, I loaded it up with Fedora Linux and set out to ensure that everything works.</p> <p>This post includes all of the configurations and changes I added along the way.</p> <h1 id="power-management" class="relative group">Power management <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#power-management" aria-label="Anchor">#</a></span></h1><p>I removed the power profiles daemon that comes with Fedora by default. and replaced it with <a href="https://linrunner.de/tlp/index.html" target="_blank" rel="noreferrer">tlp</a>. This is a great package for ThinkPad laptops as it takes care of most of the power management configuration for you with sane defaults. It also offers an easy to read configuration file where you can make adjustments.</p> <p>The defaults seem to be working well so far, but my only complaint is that the power management for <code>amdgpu</code> seems to be <em>really aggressive</em>. Graphics performance on battery power is <em>okay</em>, but I&rsquo;m told this improves in kernel 6.7. I&rsquo;m on 6.6.11 in Fedora 39 right now.</p> <p>I&rsquo;ll wait to see if this new kernel makes any improvements.</p> <h1 id="touchpad" class="relative group">Touchpad <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#touchpad" aria-label="Anchor">#</a></span></h1><p>The ELAN touchpad in the Z13 is a bit different. It&rsquo;s a <em>haptic</em> touchpad. It doesn&rsquo;t push down with a click like the other thinkpads. It provides haptic feedback, much like a mobile phone does when you tap on the screen. (I usually turn this off on my phone, but it feels good on the laptop.)</p> <p>The touchpad works right out of the box without any additional configuration. I made a basic Sway configuration stanza to get it configured with my preferences:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># ThinkPad Z13 Gen 2 AMD Touchpad</span> </span></span><span class="line"><span class="cl"><span class="na">input &#34;11311:40:SNSL0028:00_2C2F:0028_Touchpad&#34; {</span> </span></span><span class="line"><span class="cl"> <span class="na">drag disabled</span> </span></span><span class="line"><span class="cl"> <span class="na">tap enabled</span> </span></span><span class="line"><span class="cl"> <span class="na">dwt enabled</span> </span></span><span class="line"><span class="cl"> <span class="na">natural_scroll disabled</span> </span></span><span class="line"><span class="cl"><span class="na">}</span> </span></span></code></pre></div><p>The configuration above enables tap to click and dragging with taps. I like the old school scrolling style and I&rsquo;ve disabled the natural scroll.</p> <p>You can always get a list of your input devices in Sway with <code>swaymsg</code>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">swaymsg -t get_inputs </span></span></span></code></pre></div><h1 id="display" class="relative group">Display <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#display" aria-label="Anchor">#</a></span></h1><p>The display worked right out of the box but the UI elements were scaled up far too large for me. I typically value screen real estate over all other aspects, but my usual default of scaling to 1.0 made the UI far too small.</p> <p>I set my output scaling to 1.2:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="c1"># Disable HiDPI</span> </span></span><span class="line"><span class="cl"><span class="na">output * scale 1.2</span> </span></span></code></pre></div><p>I also enabled the <a href="https://rpmfusion.org/Howto/Multimedia" target="_blank" rel="noreferrer">RPM Fusion repos</a> to get the freeworld AMD Mesa drivers:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo dnf install \ </span></span></span><span class="line"><span class="cl"><span class="go"> https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \ </span></span></span><span class="line"><span class="cl"><span class="go"> https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm </span></span></span><span class="line"><span class="cl"><span class="go">sudo dnf swap mesa-va-drivers mesa-va-drivers-freeworld </span></span></span><span class="line"><span class="cl"><span class="go">sudo dnf swap mesa-vdpau-drivers mesa-vdpau-drivers-freeworld </span></span></span></code></pre></div><h1 id="audio" class="relative group">Audio <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#audio" aria-label="Anchor">#</a></span></h1><p>Sound worked right out of the box, but I found that the loudness preset from <a href="https://github.com/wwmm/easyeffects" target="_blank" rel="noreferrer">easyeffects</a> made the speakers sound a little bit better:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo dnf install easyeffects </span></span></span></code></pre></div><h1 id="everything-else" class="relative group">Everything else <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#everything-else" aria-label="Anchor">#</a></span></h1><p>Everything else just worked!</p> <p>I&rsquo;m really pleased with the performance and the battery life so far. My only complaint is that the OLED screen can be a battery hog at times.</p> <p>For more details, check out the <a href="https://wiki.archlinux.org/title/Lenovo_ThinkPad_Z13" target="_blank" rel="noreferrer">Arch Linux wiki page for the Z13</a>. They documented lots of the function keys if you want to create keyboard shortcuts and they link to some downloadable monitor profiles.</p>Dark mode in Swayhttps://major.io/p/sway-dark-mode/Tue, 09 Jan 2024 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/sway-dark-mode/<p>Ah, dark mode! I savor my dark terminals, window decorations, and desktop wallpapers. It&rsquo;s so much easier on my eyes on those long work days. 😎</p> <p>However, I think the author <a href="https://en.wikipedia.org/wiki/Mary_Oliver" target="_blank" rel="noreferrer">Mary Oliver</a> said it best:</p> <blockquote> <p>Someone I loved once gave me a box full of darkness. It took me years to understand that this too, was a gift.</p> </blockquote> <p>In most window managers, such as GNOME or KDE, switching to dark mode involves a simple trip to the settings panels and clicking different themes. Sway doesn&rsquo;t offer us those types of comforts, but we can get dark mode there, too!</p> <p> <figure> <picture class="mx-auto my-0 rounded-md" > <img width="295" height="273" class="mx-auto my-0 rounded-md" alt="sunglasses-wiggle.gif" loading="lazy" decoding="async" src="https://major.io/p/sway-dark-mode/sunglasses-wiggle.gif" /> </picture> </figure> </p> <h1 id="gtk-applications" class="relative group">GTK applications <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#gtk-applications" aria-label="Anchor">#</a></span></h1><p>If you happen to have GNOME on your system alongside sway, go into <strong>Settings</strong>, then <strong>Appearance</strong> and select <em>Dark</em>. You can also get dark mode by applying a setting in <code>~/.config/gtk-3.0/settings.ini</code>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Settings]</span> </span></span><span class="line"><span class="cl"><span class="na">gtk-application-prefer-dark-theme</span><span class="o">=</span><span class="s">1</span> </span></span></code></pre></div><p>Restart whichever application you were using and it should pick up the new configuration.</p> <p>Firefox, for example, ships with an automatic appearance setting that follows the OS. That should be reflected immediately upon restart. If not, go into Firefox&rsquo;s settings, and look for dark mode under the <strong>Language and Appearance</strong> section of the general settings.</p> <h1 id="qt-applications" class="relative group">QT applications <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#qt-applications" aria-label="Anchor">#</a></span></h1><p>Most of my applications are GTK-based, but I have one or two which use QT. Again, just like the GTK example, if you have KDE installed along side Sway, you can configure dark mode there easily. Just open the system settings and look for <em>Breeze Dark</em> in the <strong>Plasma Style</strong> section.</p> <p>You don&rsquo;t have KDE? Don&rsquo;t worry! There are a couple of commands which should work:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="c1"># This should work for all QT/KDE apps</span> </span></span><span class="line"><span class="cl"><span class="c1"># if you have the Breeze Dark theme installed.</span> </span></span><span class="line"><span class="cl">lookandfeeltool -platform offscreen <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --apply <span class="s2">&#34;org.kde.breezedark.desktop&#34;</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># You can set the theme for GTK apps here as well</span> </span></span><span class="line"><span class="cl"><span class="c1"># if you run into problems.</span> </span></span><span class="line"><span class="cl">dbus-send --session --dest<span class="o">=</span>org.kde.GtkConfig <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --type<span class="o">=</span>method_call /GtkConfig org.kde.GtkConfig.setGtkTheme <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> <span class="s2">&#34;string:Breeze-dark-gtk&#34;</span> </span></span></code></pre></div><h1 id="alternate-dark-mode-based-on-time" class="relative group">Alternate dark mode based on time <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#alternate-dark-mode-based-on-time" aria-label="Anchor">#</a></span></h1><p>Many window managers offer a method for adjusting dark and light modes based on the time of day. For example, some people love brighter interfaces during the day and darker ones at night. There&rsquo;s a great tool called <a href="https://gitlab.com/WhyNotHugo/darkman" target="_blank" rel="noreferrer">darkman</a> that makes this easier. πŸ€“</p> <p>The darkman service runs in the background and runs various commands to change dark mode settings for all kinds of window managers. It also speaks to dbus directly to set the configurations if needed.</p> <p>It also has a <a href="https://gitlab.com/WhyNotHugo/darkman/-/tree/main/examples/dark-mode.d?ref_type=heads" target="_blank" rel="noreferrer">directory full of user contributed scripts</a> to change dark and light modes for various environments. You might be able to pull some commands from these files to test which configurations might work best on your system.</p>On diversityhttps://major.io/p/on-diversity/Sat, 16 Dec 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/on-diversity/<p>️ πŸ‘‹ <em>This post represents my own views on the topic of diversity and it doesn&rsquo;t represent the views of my employer or any professional group I belong to.</em></p> <hr> <p>I&rsquo;ve written a post on diversity and deleted it several times. It remains a sensitive topic for different people for different reasons. My gut feeling is that no matter how you frame a post on diversity, some group of people will be upset about it<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p> <p>There was a great speaker who came and spoke to us at my last job and she made an excellent point that I remember today:</p> <blockquote> <p>Your experiences are yours. Nobody can take them away from you.</p> <p>Nobody can say that your experiences do not matter.</p> <p>Nobody can tell you that you didn&rsquo;t experience what you experienced.</p> <p>Sharing these experiences with others allows us to grow and understand more about the world around us.</p> </blockquote> <p>That speech entirely changed my way of thinking about interactions with other people at work and at home. There are two main benefits here:</p> <ul> <li>It&rsquo;s incredibly freeing for someone who has experienced something to be able to share it with others and not be told that their experience was wrong or misguided.</li> <li>It&rsquo;s also freeing for the listener to take in someone else&rsquo;s experience and be able to ask clarifying questions so they get a better understanding of how something felt for someone else.</li> </ul> <p>With that in mind, here&rsquo;s we go with the rest of the post. I&rsquo;m not deleting it this time.</p> <p>I promise. πŸ˜‰</p> <h1 id="my-first-experience" class="relative group">My first experience <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-first-experience" aria-label="Anchor">#</a></span></h1><p>I&rsquo;ve written in the past about my unexpected leap to lead an information security architecture team in a previous role. Being a director was a new world unto itself, but then I found that my team wasn&rsquo;t performing well. To make matters worse, our success was critical to the ongoing work of the security department as a whole.</p> <p>We had three members of the team that all brought something unique to the team&rsquo;s perspective. All three were men of different races, but each had a different approach to security based on their backgrounds and experiences. One left the team due to some interpersonal issues that eventually boiled over.</p> <p>I was suddenly down to two people and our team needed to hire two as soon as possible.</p> <p>Recruiting teams started putting feelers out into the market to find talented people and I was poking several friends for referrals. A colleague in another department reached out and really wanted to join the team. I knew about her experience from several previous interactions and she was highly recommended from her peers. She joined the team and hit the ground running.</p> <p>As applicants began trickling in through the recruiting team, I started with screening calls for each. We really needed someone with skills in a few key areas:</p> <ul> <li>Great communicator with empathy</li> <li>Knowledge of secure development and operations practices</li> <li>Someone who could be trusted to operate independently and work on team projects</li> </ul> <p>Most of the applicants I screened were male and that wasn&rsquo;t a surprise at the time. We brought five through the screening into interviews and it was down to four males and one female. Three of them turned out to be great and they all had deep knowledge of security architecture. After another round of interviews, we began to realize that the female matched the other applicants, but her communication skills were stronger, especially under pressure.</p> <p>Needless to say, we sent her the offer and she accepted! We were thrilled! Our team was full!</p> <h1 id="getting-underway" class="relative group">Getting underway <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#getting-underway" aria-label="Anchor">#</a></span></h1><p>We began chipping away at the mountain of projects set aside for our team and started making progress. Our new team member was struggling to move from the rigidity of her previous employer to our new way of working, but she adjusted well over time.</p> <p>I sat down in one of our weekly leadership meetings some time later. These meetings usually involved a round-the-horn of what&rsquo;s working well for each team, the threats on the board for the next few months, and our plans.</p> <p>We usually had an attendee from HR in the meetings for various reasons and she asked me how our new team member was doing. I said:</p> <blockquote> <p>Oh, she&rsquo;s doing a good job. Her last company was pretty rigid and things are different here, but she&rsquo;s figuring it out. She knows her stuff and she&rsquo;s a team player. We&rsquo;re working through some small things here and there.</p> </blockquote> <p>Then the HR representative said:</p> <blockquote> <p>Well, I have to commend you for building out a such a diverse team. It&rsquo;s much more so than the other teams. That&rsquo;s really great work and I want to make sure you&rsquo;re recognized for it.</p> </blockquote> <p>I smiled and thanked her (because that&rsquo;s my usual response), but then I almost felt sick.</p> <h1 id="different-view" class="relative group">Different view <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#different-view" aria-label="Anchor">#</a></span></h1><p>I left that meeting and went back to my desk to think.</p> <p>Had I assembled a diverse team intentionally? No, I didn&rsquo;t. I looked for people who had the qualities we desperately needed, gave them guardrails, and got out of the way. That&rsquo;s what you do with smart people, right?</p> <p>Then I wondered, <em>&ldquo;Is my team really diverse?&rdquo;</em></p> <ul> <li>There were two men and two women.</li> <li>Two were on the younger end of a generation and two were on the older end.</li> <li>One was of mixed Asian descent, one was Hispanic, and two were what most people would likely refer to as &ldquo;white.&rdquo;</li> </ul> <p>So maybe my team <em>is diverse</em>.</p> <p>Then I realized that most of the people on the team had the same certifications, all had at least an undergraduate degree, and all were married. All of them were in heterosexual relationships and all dressed in a way that aligned with their gender.</p> <p>Does that mean they&rsquo;re not diverse?</p> <p>Is my team more or less diverse than other teams?</p> <p>Does any of this even matter?</p> <p>Did I do a good or a bad thing?</p> <h1 id="diversity-challenges" class="relative group">Diversity challenges <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#diversity-challenges" aria-label="Anchor">#</a></span></h1><p>This brings me to the two problems I struggle with most around diversity, especially when people talk about <em>increasing</em> or <em>improving</em> diversity on their team or within their company:</p> <ol> <li>Quantifying diversity is highly subjective and in the eye of the beholder.</li> <li>Challenges arise when you apply diversity requirements to real world situations.</li> </ol> <p>I&rsquo;ll break down both of these now.</p> <h2 id="in-the-eye-of-the-beholder" class="relative group">In the eye of the beholder <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#in-the-eye-of-the-beholder" aria-label="Anchor">#</a></span></h2><p>You can choose how you want to measure diversity on all kinds of factors. Depending on the factors, a team can look more or less diverse. Also, your experiences often define how you judge the diversity of people and teams.</p> <p>One could argue that a team made up entirely of white males is likely not very diverse. The majority of people would likely agree with that statement.</p> <p>However, what if those males vary in their sexual orientations, educational backgrounds, and socioeconomic status. Is that diverse?</p> <p>If you have a team of people made up of various genders with various sexual orientations from all contents on the planet, but they all went to Ive League schools and they&rsquo;re all wealthy &ndash; is that diverse?</p> <p>Are any of these examples diverse <em>enough?</em> Does the answer to that question <em>even matter?</em></p> <p>In my experience, assembling a team of people with different backgrounds and approaches to problems is incredibly valuable. That type of diversity led to some incredible innovation in the past.</p> <p>However, these diverse backgrounds and approaches don&rsquo;t always line up with differences in gender identity, socioeconomic status, sexual orientation, or other factors. This is why I find it really challenging to quantify the level of diversity within a company or in individual teams.</p> <h2 id="rubber-meets-the-road" class="relative group">Rubber meets the road <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#rubber-meets-the-road" aria-label="Anchor">#</a></span></h2><p>There&rsquo;s a common phrase in English: <em>&ldquo;when the rubber meets the road.&rdquo;</em></p> <p>In a literal sense, it&rsquo;s referring to when car tires move on pavement during a race. What it really means is, when it comes time to do something for real and the stakes are high, what happens?</p> <p>Here&rsquo;s another example. Let&rsquo;s say you lead an engineering team that is all males and your company says that diversity must be a priority in hiring decisions.</p> <p>So you take your job requisition and send it through the recruiting team. They work hard to remove any gender-specific language or anything else that might turn an applicant away. You put the job on the internet, talk to your friends about people they know, and then wait for the responses.</p> <p>Let&rsquo;s assume you get ten male applicants.</p> <p>Do you proceed with screening and interviewing them while you try harder to drum up more female applicants? If a female applicant never appears, do you pause the hiring process while you try to find one? What if your existing applicants find other roles in the meantime and suddenly your applicant pipeline is empty?</p> <p>Some might say <em>&ldquo;Yes, of course you wait until you can find a female applicant!&rdquo;</em> In that case, your team is still short-staffed and likely not performing as well as it could. Would that be good for your customers? How about your shareholders?</p> <p>Others might say <em>&ldquo;No, go ahead and complete the hiring process but you should search harder for women for future roles.&rdquo;</em> In this case, you&rsquo;ll have a fully staffed team and hopefully be delivering more value quickly. However, you haven&rsquo;t improved the diversity on your team and that could come back to be a problem if you&rsquo;re asked about it later.</p> <hr> <p>Go backwards a bit with the same example and assume you get a split of ten applicants: half male and half female. That&rsquo;s awesome because now you have a diverse talent pool, right?</p> <p>Here&rsquo;s where it gets challenging.</p> <p>If you interview them all and make an offer to the female applicant because she has the skills and qualifications needed, you now have a more diverse team (on one measure) and you&rsquo;re fully staffed! Great!</p> <p>If you interview them all and it turns out one of the men has the best skills and qualifications, what do you do? Your company made diversity a priority, but you&rsquo;re also trying to assemble a strong team.</p> <p>Do you take a less qualified applicant that improves the team&rsquo;s diversity?</p> <p>Or, do you take a more qualified applicant that leaves the team&rsquo;s diversity unchanged?</p> <p>This is where diversity breaks down: when you have to really sit down and compare outcomes, there&rsquo;s not a right answer.</p> <h1 id="another-viewpoint" class="relative group">Another viewpoint <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#another-viewpoint" aria-label="Anchor">#</a></span></h1><p>My wife constantly points out things to me that I completely missed and we&rsquo;ve talked about this topic many times. She has asked me the same thing in the past:</p> <blockquote> <p>Why do people in your field care so much about getting women into technology? I hate technology. Maybe other women hate technology, too. If I knew I was hired someplace because they wanted a woman for the role and they weren&rsquo;t looking at how well someone could do the job, I&rsquo;d be pretty upset.</p> </blockquote> <p>She&rsquo;s a medical professional and she&rsquo;s happy to remind me about this:</p> <blockquote> <p>I went to PA (physician assistant) school and most people there were women. All the nurses at my office are women. All the front office staff are women. We&rsquo;re not out there trying to get male nurses or male front office staff in here all the time. We just find people who do their job well and hire them.</p> </blockquote> <p>Our conversations really make me stop and think.</p> <h1 id="my-goals" class="relative group">My goals <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-goals" aria-label="Anchor">#</a></span></h1><p>I&rsquo;d also like to see more people from underrepresented communities across the globe break into the world of technology and really change things. This means empowering a wider array of people with varying gender, education, nationality, wealth, and opportunities to join a field of work which they thought might be inaccessible to them.</p> <p>This is why I try to volunteer as much as possible to inspire young people of all backgrounds to set goals for themselves and look at the world as if nothing is out of reach.</p> <p>It&rsquo;s one of the reasons I write this blog and put everything out there for free. Democratizing access to learning (and my mediocre blog posts) is key to leveling the playing field.</p> <p>These are some of those pieces of work that are never finished.</p> <p>However, I really worry that quantifying diversity or forcing one&rsquo;s definition of diversity onto someone else could lead us to a bad place where no result is satisfactory. It&rsquo;s much more subjective than some would like to admit and that becomes a problem when you directly apply it to specific situations.</p> <p>In the meantime, I&rsquo;ll keep writing these posts, mentoring others, and lifting people up to do things they never imagined they could do. ️β™₯️</p> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>Then again, we live in a world where someone can say <em>&ldquo;Puppies are cute&rdquo;</em> and the first reply would be <em>&ldquo;Why do you hate cats so much?&rdquo;</em> πŸ˜„&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>Horror book reviews from October 2023https://major.io/p/horror-book-reviews/Sun, 19 Nov 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/horror-book-reviews/<p>Reading allows me to travel to other places and times while also reducing my stress and helping me to think more creatively. Sometimes this leads me to wild fictional stories or takes me on a learning journey into history.</p> <p><em>(I <a href="https://www.goodreads.com/user/show/49133137-major-hayden" target="_blank" rel="noreferrer">track my reading lists on Goodreads</a> if you want to see what I&rsquo;m reading.)</em></p> <p>In this post, I&rsquo;ll list the spooky books I read this October and hopefully you&rsquo;ll find at least one them interesting!</p> <h1 id="the-cabin-at-the-end-of-the-world" class="relative group">The Cabin at the End of the World <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#the-cabin-at-the-end-of-the-world" aria-label="Anchor">#</a></span></h1><p>My first book of the month was Paul Tremblay&rsquo;s <a href="https://www.goodreads.com/book/show/36381091-the-cabin-at-the-end-of-the-world" target="_blank" rel="noreferrer">The Cabin at the End of the World</a>. It&rsquo;s centered around a family with a small child that goes on a relaxing vacation in a lakefront cabin.</p> <p>All is well until a friendly stranger named Leonard befriends the child, Wen, and his three fellow travelers appear. The travelers hold the family hostage and tell them that they have the key to save the world, but it&rsquo;s not as easy as it seems. There seems to be an unseen force that has some sort of control over the travelers. πŸ€”</p> <h2 id="my-thoughts" class="relative group">My thoughts <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-thoughts" aria-label="Anchor">#</a></span></h2><p>I thought I had this story figured out several times only to be proven wrong just as many times. There are so many themes in this book that tug at your emotions, including family, racism, homophobia, and socioeconomic differences. This book packs plenty of suspense, but the most frightening parts aren&rsquo;t supernatural or ghostly. The scariest parts feel centered around the essence of human nature.</p> <p>Although this felt like a quick read, it was intense in many places. I definitely enjoyed it and I was looking forward to seeing the movie adaptation, <a href="https://en.wikipedia.org/wiki/Knock_at_the_Cabin" target="_blank" rel="noreferrer">Knock at the Cabin</a>. The movie was done by <a href="https://en.wikipedia.org/wiki/Knock_at_the_Cabin" target="_blank" rel="noreferrer">M. Night Shyamalan</a> (of <a href="https://en.wikipedia.org/wiki/The_Sixth_Sense" target="_blank" rel="noreferrer">The Sixth Sense</a> fame) and I read that he changed the entire ending of the movie. 😞</p> <h1 id="the-ruins" class="relative group">The Ruins <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#the-ruins" aria-label="Anchor">#</a></span></h1><p>I moved onto something <em>completely different</em> next with Scott Smith&rsquo;s <a href="https://www.goodreads.com/book/show/21726.The_Ruins" target="_blank" rel="noreferrer">The Ruins</a>. The story takes place in Mexico with several people on a beach vacation. One of the tourists notes that his brother went to check out an archeological dig further into the countryside but never returned. The group eventually decides to go search for the missing brother as some sort of vacation adventure.</p> <p>The adventure turns ugly as they make their way to the country&rsquo;s interior. 😬</p> <h2 id="my-thoughts-1" class="relative group">My thoughts <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-thoughts-1" aria-label="Anchor">#</a></span></h2><p>You&rsquo;ll have a tough time finding the antagonist in this story until it&rsquo;s too late, but that&rsquo;s part of the fun. Every character in the group brings personality quirks and old baggage with them that impairs their judgement in different ways. This works well for some but not for others.</p> <p>This book had several scary moments, but most of the horror came again from how humans interact with one another. As soon as someone (or something) else figures out how to exploit those against them, bad things happen.</p> <p>This book was difficult to read because much of it was quite gruesome and brutal. It&rsquo;s definitely a book for adults only and you should be prepared to work your way through it.</p> <h1 id="the-troop" class="relative group">The Troop <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#the-troop" aria-label="Anchor">#</a></span></h1><p>Everything is fine with a scout trip to an island in Canada in Nick Cutter&rsquo;s <a href="https://www.goodreads.com/book/show/17571466-the-troop" target="_blank" rel="noreferrer">The Troop</a>. Well, it&rsquo;s fine until a mysterious and incredibly hungry man suddenly shows up on the island. He looks a lot like he&rsquo;s dead already and the the leader of the scout troop, a medical doctor, is completely mystified by the man&rsquo;s condition.</p> <p>It goes downhill from there in a story told mostly through diary entries, newspaper clippings, and court testimony.</p> <h2 id="my-thoughts-2" class="relative group">My thoughts <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-thoughts-2" aria-label="Anchor">#</a></span></h2><p>Of all the books I read in October, this one scared me the most because it felt like it was entirely possible. There wasn&rsquo;t any part of the book that I looked at and said: &ldquo;Oh, that could never happen.&rdquo; That&rsquo;s what makes this one so good.</p> <p>It&rsquo;s suspenseful enough to keep you turning the pages but it&rsquo;s also plausible enough that you might find yourself wanting to wash your hands a little longer the next time you eat a meal. It feels a bit like <a href="https://www.goodreads.com/book/show/7624.Lord_of_the_Flies" target="_blank" rel="noreferrer">Lord of the Flies</a> mixed with a pandemic novel like <a href="https://www.goodreads.com/book/show/20170404-station-eleven" target="_blank" rel="noreferrer">Station Eleven</a> or <a href="https://www.goodreads.com/book/show/87591651-the-stand" target="_blank" rel="noreferrer">The Stand</a>.</p> <p>This one is also <em>very gruesome</em> in parts with some very difficult scenes to read.</p> <p>This one was my favorite of the group by far.</p> <h1 id="devolution-a-firsthand-account-of-the-rainier-sasquatch-massacre" class="relative group">Devolution: A Firsthand Account of the Rainier Sasquatch Massacre <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#devolution-a-firsthand-account-of-the-rainier-sasquatch-massacre" aria-label="Anchor">#</a></span></h1><p>Sasquatch is back in Max Brooks&rsquo; <a href="https://www.goodreads.com/book/show/52454426-devolution" target="_blank" rel="noreferrer">Devolution</a>! Told through the journals of a woman living in a remote town in Washington, this book covers a modern time where Mount Ranier erupted and caused lots of species to get on the move from their habitats around the mountain. Some of those creatures are the ones you&rsquo;d expect, but some are ones you won&rsquo;t expect.</p> <p>The small community is an experiment in green, off the grid living, and they are quickly tested by just about everything mother nature can throw at them. This includes some rather tall, furry, human-like creatures.</p> <h2 id="my-thoughts-3" class="relative group">My thoughts <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-thoughts-3" aria-label="Anchor">#</a></span></h2><p>This one felt scary due to the remoteness of the village and the unprepared people involved. Also, whether you believe in Sasquatch or not, this felt a bit more plausible than I expected.</p> <p>There were plenty of difficult to read scenes in here, but the gore was reduced compared to other books listed here. Much of the suspense came from how humans interact with one another especially when faced with an adversary that presents a unique set of challenges.</p> <p>This book was difficult to get into (it starts slow), but stick with it. It&rsquo;s a wild ride.</p> <h1 id="tender-is-the-flesh" class="relative group">Tender is the Flesh <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#tender-is-the-flesh" aria-label="Anchor">#</a></span></h1><p>Be prepared when you crack open Agustina Bazterrica&rsquo;s <a href="https://www.goodreads.com/book/show/49090884-tender-is-the-flesh" target="_blank" rel="noreferrer">Tender is the Flesh</a>. Imagine a world where animals somehow contract a virus that they quickly spread to each other and to humans as well. Pets are banned and animals near cities are killed. The pandemic puts a significant dent in the human population.</p> <p>However, with all of the animals either infected or gone, where do people get meat to eat? πŸ€”</p> <h2 id="my-thoughts-4" class="relative group">My thoughts <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-thoughts-4" aria-label="Anchor">#</a></span></h2><p>This was by far the most challenging book to read out of the group because I honestly felt like I was going to be sick in places. The author doesn&rsquo;t set out for cheap scares or basic unsettling events &ndash; there&rsquo;s something much deeper. It really makes you question how humanity operates and how quickly boundaries can shift when hunger becomes a problem.</p> <p>I had to take a lot of breaks with this book. It has some suspenseful parts but this book has a slow-burn horror feel that gives you hope, crushes that hope, and then starts the cycle once more.</p> <p>I strongly recommend this book <em>for adults only</em> but I feel terrible recommending it at the same time. πŸ˜„</p> <h1 id="whats-next" class="relative group">What&rsquo;s next? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#whats-next" aria-label="Anchor">#</a></span></h1><p>After all of that horror and unsettling fiction, I&rsquo;m shaking things up for November. My current book is Larry McMurtry&rsquo;s <a href="https://www.goodreads.com/book/show/256008.Lonesome_Dove" target="_blank" rel="noreferrer">Lonesome Dove</a>. So many people have recommended it to me as a great book about the American west and I&rsquo;m enjoying it so far.</p>Moving to cloud is more than just a purchasing exercisehttps://major.io/p/cloud-more-than-purchasing-exercise/Fri, 27 Oct 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/cloud-more-than-purchasing-exercise/<p>Much of my work at Red Hat revolves around the RHEL experience in public clouds. I thrive on input from customers, partners, and coworkers about how they consume public clouds and why they made decisions to deploy there.</p> <p>Throughout this process, I run into some wild misconceptions about public clouds and what makes them useful. One that I hear most often is:</p> <blockquote> <p>Businesses are moving to the cloud to reduce cost and improve efficiency. It&rsquo;s mainly just a purchasing exercise.</p> </blockquote> <p>This couldn&rsquo;t be further from the truth.</p> <h1 id="cloud-offers-a-chance-to-start-over" class="relative group">Cloud offers a chance to start over <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#cloud-offers-a-chance-to-start-over" aria-label="Anchor">#</a></span></h1><p>Sometimes businesses find themselves in an IT quagmire<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. No matter what they do to improve their situation, it just gets worse. Capital expenditures grow and grow, datacenter space gets more expensive, and companies spend more time focusing on IT rather than their core business.</p> <p>Deploying in clouds offers that chance to break the capital expense cycle and gradually improve infrastructure. The key word here is <em>gradual</em>.</p> <p>Businesses can choose how much they want to deploy and when without worrying about expensive servers in the datacenter waiting to be used. Some deployments are greenfield, or entirely net new applications. Some are basic migrations of applications from servers or virtual machines directly to the cloud.</p> <p>Either way, businesses have the freedom to deploy as little or as much as they want on their own schedule.</p> <h1 id="cloud-offers-a-chance-to-software-define-nearly-anything" class="relative group">Cloud offers a chance to software-define (nearly) anything <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#cloud-offers-a-chance-to-software-define-nearly-anything" aria-label="Anchor">#</a></span></h1><p>Anyone who has worked in a large organization before knows the pain of change management. Sure, it ticks a box on that yearly compliance program, but it also ensures that everyone is aligned on the plan.</p> <p>One of the greatest aspects of cloud is that you can define almost everything in software. This makes changes easier to apply, easier to roll back, and easier to track.</p> <p>Tools like <a href="https://www.terraform.io/" target="_blank" rel="noreferrer">Terraform</a> or <a href="https://www.ansible.com/" target="_blank" rel="noreferrer">Ansible</a> allow developers and operations team to work from the same playbook. My team enjoys using <a href="https://www.infracost.io/" target="_blank" rel="noreferrer">Infracost</a> to track how much a particular Terraform change might cost us under different scenarios.</p> <p>Once teams set a policy of &ldquo;we define our changes in git, and that&rsquo;s it&rdquo;, you can rely on a git history for change management. This avoids drift in production environments and it also ensures that changes made in development environments make it into staging and then into production. The days of <em>&ldquo;it worked on my system, what&rsquo;s wrong with production?&rdquo;</em> slowly fade away.</p> <p>Less than ideal architectural decisions can also be adjusted over time to fit the applications being deployed. Did you set up a network incorrectly? Did you choose an instance type without enough RAM?</p> <p>That&rsquo;s okay!</p> <p>Just adjust the deployment in git, test it in staging, and push it to production.</p> <h1 id="cloud-offers-managed-services" class="relative group">Cloud offers managed services <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#cloud-offers-managed-services" aria-label="Anchor">#</a></span></h1><p>One thing I tell people constantly is that if you bend the cloud to fit your application, <strong>you will almost always pay more.</strong> You get cost and performance efficiencies if you bend your application to fit the cloud. Confused? I&rsquo;ll explain.</p> <p>When I talk to people doing their first cloud deployments, they deploy everything into VMs, much as they would in a local virtualized environment.</p> <ul> <li><em>You need a database server?</em> Make a couple of VMs and set up replication.</li> <li><em>You need to run a batch job via cron?</em> Deploy a VM and add it to the crontab.</li> <li><em>You need a server to export an NFS share?</em> Deploy a VM with lots of storage and export it to other instances.</li> </ul> <p>Do you see a pattern here?</p> <p> <figure> <picture class="mx-auto my-0 rounded-md" > <img width="480" height="270" class="mx-auto my-0 rounded-md" alt="too-much.gif" loading="lazy" decoding="async" src="https://major.io/p/cloud-more-than-purchasing-exercise/too-much.gif" /> </picture> </figure> </p> <p>Most public clouds offer tons of services that lift the management burden from engineering teams and offload into a managed service. For example, that cron job might be able to move into a &ldquo;serverless&rdquo;<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> service, such as <a href="https://aws.amazon.com/lambda/" target="_blank" rel="noreferrer">AWS Lambda</a>. It&rsquo;s critical to check the pricing here to ensure you&rsquo;re not headed down a bad path, but you have one less VM to maintain, one less IPv4 address to pay for, and a greatly reduced risk of configuration drift. That reduction in stress and risk might be worth any additional costs.</p> <p>Deployment decisions become much easier and lower stress when you consume services offered by the provider. There are those situations where deploying a whole VM is needed, but I&rsquo;ve managed to avoid that for some of my team&rsquo;s recent deployments.</p> <p>Our last deployment uses GitHub Actions, S3, and CloudFront and costs us about $6.50 per month to run. There are no virtual machines. There&rsquo;s nothing to patch.</p> <p>This blog <a href="https://major.io/p/cloudfront-migration/">runs on a similar stack</a> and costs me about $0.25 per month to run.</p> <h1 id="cloud-offers-geographic-distribution" class="relative group">Cloud offers geographic distribution <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#cloud-offers-geographic-distribution" aria-label="Anchor">#</a></span></h1><p>Nearly every public cloud, even the smallest ones, offer you the same or similar services in a wide variety of geographic regions. Disaster recovery feels more attainable when you can easily deploy to multiple regions with the same software-defined infrastructure.</p> <p>Data sovereignty continues to grow in importance around the world as more countries demand that their data remains within their borders. As long as your cloud offers a region in that country, you can deploy there. There&rsquo;s no challenging legal issues with finding datacenter space or getting hardware delivered. You just change your region and deploy.</p> <p>Cloud regions also allow you to bring your applications much closer to the people who use them. Reduced latency delivers content faster to customers and provides a responsive experience.</p> <h1 id="clouds-offer-purchasing-efficiency" class="relative group">Clouds offer purchasing efficiency <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#clouds-offer-purchasing-efficiency" aria-label="Anchor">#</a></span></h1><p><strong>Wait a minute!</strong> Didn&rsquo;t I say that moving to cloud isn&rsquo;t just a purchasing exercise? πŸ€”</p> <p> <figure> <picture class="mx-auto my-0 rounded-md" > <img width="480" height="281" class="mx-auto my-0 rounded-md" alt="surprise.gif" loading="lazy" decoding="async" src="https://major.io/p/cloud-more-than-purchasing-exercise/surprise.gif" /> </picture> </figure> </p> <p>Your move to cloud should not be solely based on cutting costs or making purchasing IT more efficient. Most teams find that moving to cloud is more expensive than they anticipated because they&rsquo;re finally able to get access to the right amount of resources that they need. <em>(Also, they usually go with some more expensive options up front until they figure out how to optimize for cost.)</em></p> <p>First off, it&rsquo;s much easier to budget and pay one vendor for multiple services than deal with multiple independent vendors. Instead of paying for datacenter space, then paying for servers, then paying for network equipment, then paying for people to set it up, and so on, you pay the cloud provider for all of it.</p> <p>This also extends to other purchases on the cloud, such as products from certain vendors. For example, you can buy Red Hat products directly from some cloud providers and that gets added onto your cloud invoice. You can even deploy your own <a href="https://aws.amazon.com/marketplace/pp/prodview-sltshxd3bzqbg" target="_blank" rel="noreferrer">Cisco ASA in the cloud</a> if you feel so inclined.</p> <p>With all of these purchases going through one vendor, you can also negotiate discounts if you set a spending commitment. Discounts depend on your committed spend, of course, and the term that you agree to spend it. There&rsquo;s a whole industry around financial operations in the cloud, called <a href="https://www.finops.org/introduction/what-is-finops/" target="_blank" rel="noreferrer">FinOps</a>, and this is one of many things that factors into it.</p> <h1 id="wrapping-up" class="relative group">Wrapping up <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#wrapping-up" aria-label="Anchor">#</a></span></h1><p>Public clouds offer an incredible amount of opportunity to get your IT deployments into better shape with better change control and a solid software-defined workflow. They also offer the ability to &ldquo;write one check&rdquo; to consume infrastructure via utility billing.</p> <p><strong>However, public clouds are not ideal for every application or situation.</strong></p> <p>Do I think that <strong>every company</strong> in the world could benefit from getting some part of their IT deployments into a public cloud platform? <strong>Yes, I do.</strong></p> <p>Would <strong>every company</strong> benefit from putting most of their infrastructure into public clouds? <strong>Very unlikely.</strong></p> <p>Some applications still benefit from being on purpose-built hardware or in certain locations where a cloud might not exist today. Clouds can also be extremely expensive if you run large workloads around the clock. They can also be painful for applications with very strict or special requirements that don&rsquo;t fit a cloud deployment model well.</p> <p>The vendors that will succeed the most in the cloud space are the ones that look beyond purchasing efficiencies and IT acquisition concerns. Simply dragging the old world of physical servers or virtual machines into cloud won&rsquo;t lead anywhere.</p> <p>Those companies that help their customers <strong>benefit from the best of what public clouds have to offer</strong> in the most secure, reliable, and simple ways will be in the driver&rsquo;s seat.</p> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>A quagmire is something that gets worse no matter how you try to improve it. The only way to win is to avoid it entirely.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:2"> <p>Boy, I still dislike that <em>serverless</em> term so much. πŸ€¦β€β™‚οΈ&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>How I learned to stop worrying and love the CoreOShttps://major.io/p/why-coreos/Fri, 13 Oct 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/why-coreos/<hr> <p>It&rsquo;s quite clear that I&rsquo;ve been on a <a href="https://major.io/tags/coreos/">CoreOS</a> blogging streak lately. I keep getting asked by people inside and outside my company about what makes CoreOS special and why I&rsquo;ve switched over so many workloads to it.</p> <p>The answer is pretty basic. <strong>It makes my life easier.</strong></p> <p>I&rsquo;m a Dad. I&rsquo;m on the PTC (Parent Teacher Club) at one of my children&rsquo;s schools. I volunteer as an IT person for a non-profit. I write software. I have other time consuming hobbies, such as ham radio, reading, and becoming a longer distance runner<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p> <p>My available time for my own IT projects is <strong>extremely limited</strong> and CoreOS plays a part in keeping that part of my life as efficient as possible.</p> <p>That&rsquo;s what this blog post is about!</p> <h1 id="updates" class="relative group">Updates <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#updates" aria-label="Anchor">#</a></span></h1><p>First and foremost, I love how CoreOS does updates. I encourage you to <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/auto-updates/" target="_blank" rel="noreferrer">read the docs</a> on this topic, but here&rsquo;s a short explanation:</p> <ol> <li>Updates are automatically retrieved and they&rsquo;re loaded into a slot.</li> <li>Your system reboots into the new update but your original OS tree remains in place.</li> <li>Did the update boot? Awesome. You&rsquo;re good to go.</li> <li>Did something break? The system reverts back to the known good tree.</li> </ol> <p>In this way, it&rsquo;s a lot like your smartphone.</p> <p>You have full control over when a node looks for an update and how often it checks for them. Check out the <a href="https://coreos.github.io/zincati/usage/updates-strategy/" target="_blank" rel="noreferrer">Zincati docs</a> for tons of controls over updates and reboots.</p> <p>Some of mine are timed so well that I set maintenance windows with my monitoring provider when I know an update might take place. The updates come through, monitoring shuts off, the node reboots, and monitoring comes back. The nodes almost always come back before the monitoring even alerts me.</p> <p>It also removes the reminders I would set for myself to update packages and run reboots. I know that my CoreOS nodes will do this automatically, so I don&rsquo;t need to think about it.</p> <p>Also, updates are rarely ever impactful to my workloads since all of them are running inside containers. My containers come right back up as soon as the node finishes its reboot.</p> <h1 id="toolbox" class="relative group">Toolbox <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#toolbox" aria-label="Anchor">#</a></span></h1><p>To be fair, you can get toolbox running on lots of different Linux distributions outside of CoreOS, but that&rsquo;s the first place I ever used it. Toolbox, also called <a href="https://containertoolbx.org/" target="_blank" rel="noreferrer">toolbx</a>, gives you a utility container on your CoreOS node for all kinds of adminsitrative and diagnostic capabilities.</p> <p>You might need a certain package for diagnosting a hardware issue or you might want to install some helpful utilities for the command line. Do that in a toolbox container. Just run <code>toolbox enter</code> and if you&rsquo;ve never created a toolbox container before, you&rsquo;ll get a Fedora container that matches your CoreOS release.</p> <p>But it gets better.</p> <p>Toolbox automatically saves your container when you&rsquo;re done with it so all of your installed packages stay there for next time. Also, these containers have seamless access to anything you have in your home directory, including sockets. You&rsquo;re running inside a container, but it&rsquo;s almost like you&rsquo;re running on the host itself inside your home directory. You get the best of both worlds.</p> <p>Don&rsquo;t want Fedora? You have lots of distribution options through toolbox. Read the details on <a href="https://containertoolbx.org/install/" target="_blank" rel="noreferrer">custom images</a> to create your own!</p> <h1 id="layering" class="relative group">Layering <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#layering" aria-label="Anchor">#</a></span></h1><p>Okay, there are those situations where you really want a package on CoreOS and toolbox might not be sufficient. My muscle memory for <code>vim</code> is so strong and CoreOS only comes with <code>vi</code>.</p> <p>You have a couple of options here:</p> <ul> <li>Run <code>sudo rpm-ostree install vim</code>, reboot, and you have <code>vim</code></li> <li>Run <code>sudo rpm-ostree install --apply-live vim</code> and you have <code>vim</code> right now! <em>(And it&rsquo;s there after a reboot as well.)</em></li> </ul> <p>When a new update comes down for the base OS from CoreOS, any packages you&rsquo;ve added will be layered on the base image and available after a reboot. Layering is generally chosen as a last resort option for adding packages to the system but you shouldn&rsquo;t run into issues if you&rsquo;re installing small utilities or command line tools.</p> <h1 id="declarative-provisioning" class="relative group">Declarative provisioning <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#declarative-provisioning" aria-label="Anchor">#</a></span></h1><p>If you&rsquo;ve provisioned Linux distributions on cloud instances in the past, you&rsquo;ve likely provided metadata that cloud-init uses to provision your system. CoreOS has something that acts a lot earlier in the boot process and has more power to get things done: <a href="https://coreos.github.io/ignition/" target="_blank" rel="noreferrer">ignition</a>.</p> <p>There&rsquo;s a handy <a href="https://coreos.github.io/butane/" target="_blank" rel="noreferrer">butane</a> file forma that you use for writing your configuration. You use the <code>butane</code> utility to get it into ignition format. The ignition format is highly compressed to ensure you can fit your configuration into most cloud providers&rsquo; metadata fields.</p> <p>For a real example of what you can do with ignition, check out my <a href="https://major.io/p/quadlets-replace-docker-compose/">quadlets post</a> where I provisioned an entire Wordpress container stack using a single ignition file.</p> <p>There&rsquo;s lots of documentation for <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/" target="_blank" rel="noreferrer">writing butane configuration</a> files for common situations. It&rsquo;s easy to add files, configure Wireguard, set up users, and launch containers immediately on the first boot.</p> <h1 id="pets-and-cattle" class="relative group">Pets and cattle <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#pets-and-cattle" aria-label="Anchor">#</a></span></h1><p>CoreOS works well for systems that I only need online for a short time. These might be situations where I need to test a few containers and throw it away. There&rsquo;s no OS to mess with and no updates to worry about.</p> <p>It also works well for systems that I keep online for a long time. I have a few physical systems at home that run CoreOS and they&rsquo;ve been extremely stable. I also have cloud instances on Hetzner, VULTR, and Digital Ocean that have run CoreOS for months without issues.</p> <h1 id="more-questions" class="relative group">More questions? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#more-questions" aria-label="Anchor">#</a></span></h1><p>Feel free to <a href="mailto:major&#43;coreos@mhtx.net">send me an email</a> or <a href="https://social.lol/@major" target="_blank" rel="noreferrer">drop me a toot on Mastodon</a>. I&rsquo;ll update this post if I get some good ones!</p> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>Completing a half marathon without keeling over is the current goal! πŸ‘Ÿ&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>Quadlets might make me finally stop using docker-composehttps://major.io/p/quadlets-replace-docker-compose/Mon, 25 Sep 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/quadlets-replace-docker-compose/<hr> <p>I&rsquo;ve <a href="https://major.io/tags/containers/">written a lot about containers</a> on this blog. Why do I love containers so much?</p> <ul> <li>They start quickly</li> <li>They make your workloads portable</li> <li>They disconnect your application stack from the OS that runs underneath</li> <li>You can send your application through CI as a single container image</li> <li>You can isolate workloads on the network and limit their resource usage much like a VM</li> </ul> <p>However, I&rsquo;m still addicted to <a href="https://docs.docker.com/compose/" target="_blank" rel="noreferrer">docker-compose</a>. Can podman&rsquo;s <a href="https://www.redhat.com/sysadmin/quadlet-podman" target="_blank" rel="noreferrer">quadlets</a> change that?</p> <p><strong>Yes, I think they can.</strong></p> <h1 id="whats-a-quadlet" class="relative group">What&rsquo;s a quadlet? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#whats-a-quadlet" aria-label="Anchor">#</a></span></h1><p>Podman introduced support for quadlets in version 4.4 and it&rsquo;s a simpler way of letting systemd manage your containers. There was an option in the past to have podman generate systemd unit files, but those were unwieldy and full of podman command line options inside a unit file. These unit files weren&rsquo;t easy to edit or even parse with eyeballs.</p> <p>Quadlets make this easier by giving you a simple ini-style file that you can easily read and edit. This blog post will include some quadlets later, but here&rsquo;s an example one for Wordpress:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span> </span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Wordpress Quadlet</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Container]</span> </span></span><span class="line"><span class="cl"><span class="na">Image</span><span class="o">=</span><span class="s">docker.io/library/wordpress:fpm</span> </span></span><span class="line"><span class="cl"><span class="na">ContainerName</span><span class="o">=</span><span class="s">wordpress</span> </span></span><span class="line"><span class="cl"><span class="na">AutoUpdate</span><span class="o">=</span><span class="s">registry</span> </span></span><span class="line"><span class="cl"><span class="na">EnvironmentFile</span><span class="o">=</span><span class="s">/home/core/.config/containers/containers-environment</span> </span></span><span class="line"><span class="cl"><span class="na">Volume</span><span class="o">=</span><span class="s">wordpress.volume:/var/www/html</span> </span></span><span class="line"><span class="cl"><span class="na">Network</span><span class="o">=</span><span class="s">wordpress.network</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Service]</span> </span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">always</span> </span></span><span class="line"><span class="cl"><span class="na">TimeoutStartSec</span><span class="o">=</span><span class="s">900</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Install]</span> </span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">caddy.service multi-user.target default.target</span> </span></span></code></pre></div><p>Lots of the lines under <code>[Container]</code> should look familiar to most readers who have worked with containers before. However, there&rsquo;s something new here.</p> <p>Check out the <code>AutoUpdate=registry</code> line. This tells podman to keep your container updated on a regular basis with the upstream container registry. I&rsquo;ve used <a href="https://major.io/p/podman-quadlet-watchtower/">watchtower</a> in the past for this, but it requires a privileged container and it&rsquo;s yet another external dependency.</p> <p>Also, at the very end, you&rsquo;ll see a <code>WantedBy</code> line. This is a great place to set up container dependencies. In this example, the container that runs <code>caddy</code> (a web server) can&rsquo;t start until Wordpress is up and running.</p> <h1 id="so-why-not-stick-with-docker-compose" class="relative group">So why not stick with docker-compose? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#so-why-not-stick-with-docker-compose" aria-label="Anchor">#</a></span></h1><p>There&rsquo;s no denying that docker-compose is an awesome tool. You specify the desired outcome, tell it to bring up containers, and it gets containers into the state you specified. It handles volumes, networks, and complicated configuration without a lot of legwork. The YAML files are pretty easy to read, too.</p> <p>However, as with watchtower, that&rsquo;s another external dependency.</p> <p>My container deployments are often done at instance boot time and I don&rsquo;t make too many changes afterwards. I found myself using docker-compose for the initial deployment and then I didn&rsquo;t really use it again.</p> <p>Why not remove it entirely and use what&rsquo;s built into CoreOS already?</p> <h1 id="quaint-quadlets-quickly" class="relative group">Quaint quadlets quickly! <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#quaint-quadlets-quickly" aria-label="Anchor">#</a></span></h1><p>Before we start, we&rsquo;re going to need a few things:</p> <ul> <li>An easy to read <a href="https://coreos.github.io/butane/" target="_blank" rel="noreferrer">butane</a> configuration which gets transformed into a tiny <a href="https://coreos.github.io/ignition/" target="_blank" rel="noreferrer">ignition</a> configuration for CoreOS</li> <li>Some quadlets</li> <li>Extra system configuration</li> <li>A cloud provider with CoreOS images <em>(using <a href="https://www.vultr.com/?ref=9544589-8H" target="_blank" rel="noreferrer">VULTR</a> for this)</em></li> </ul> <p>I&rsquo;ve packed all of these items into my <a href="https://github.com/major/quadlets-wordpress" target="_blank" rel="noreferrer">quadlets-wordpress</a> repository to make it easy. Start by looking at the <a href="https://github.com/major/quadlets-wordpress/blob/main/config.butane" target="_blank" rel="noreferrer">config.butane</a> file.</p> <p>Let&rsquo;s break it down here. First up, we add an ssh key for the default <code>core</code> user.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">variant</span><span class="p">:</span><span class="w"> </span><span class="l">fcos</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="m">1.5.0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">passwd</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">users</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ssh_authorized_keys</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0</span><span class="w"> </span></span></span></code></pre></div><p>Next up, we enable the <code>podman-auto-update.timer</code> so we get container updates automatically:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">storage</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">links</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/systemd/user/timers.target.wants/podman-auto-update.timer</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">target</span><span class="p">:</span><span class="w"> </span><span class="l">/usr/lib/systemd/user/podman-auto-update.timer</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span></code></pre></div><p>Next is the long <code>files</code> section:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w"> </span><span class="nt">files</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Ensure the `core` user can keep processes running after they&#39;re logged out.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/var/lib/systemd/linger/core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="m">0644</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Allow caddy to listen on 80 and 443.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Allow it to ask for bigger network buffers, too.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/sysctl.d/90-caddy.conf</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">inline</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> net.ipv4.ip_unprivileged_port_start = 80 </span></span></span><span class="line"><span class="cl"><span class="sd"> net.core.rmem_max=2500000 </span></span></span><span class="line"><span class="cl"><span class="sd"> net.core.wmem_max=2500000</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Set up an an environment file that containers can read to configure themselves.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/containers-environment</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">inline</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> MYSQL_DATABASE=wordpress </span></span></span><span class="line"><span class="cl"><span class="sd"> MYSQL_USER=wordpress </span></span></span><span class="line"><span class="cl"><span class="sd"> MYSQL_ROOT_PASSWORD=mariadb-needs-a-secure-password </span></span></span><span class="line"><span class="cl"><span class="sd"> MYSQL_PASSWORD=wordpress-needs-a-secure-password </span></span></span><span class="line"><span class="cl"><span class="sd"> WORDPRESS_DB_HOST=mariadb </span></span></span><span class="line"><span class="cl"><span class="sd"> WORDPRESS_DB_USER=wordpress </span></span></span><span class="line"><span class="cl"><span class="sd"> WORDPRESS_DB_PASSWORD=wordpress-needs-a-secure-password </span></span></span><span class="line"><span class="cl"><span class="sd"> WORDPRESS_DB_NAME=wordpress</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="m">0644</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Deploy the caddy configuration file from the repository.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/caddy/Caddyfile</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">local</span><span class="p">:</span><span class="w"> </span><span class="l">caddy/Caddyfile</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="m">0644</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Add some named volumes for caddy and wordpress.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/systemd/caddy-config.volume</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">inline</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> [Volume]</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/systemd/caddy-data.volume</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">inline</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> [Volume]</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/systemd/wordpress.volume</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">inline</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> [Volume]</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Create a network for all the containers to use and enable the</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># DNS plugin. This allows containers to find each other using</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># the container names.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/systemd/wordpress.network</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">inline</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> [Network] </span></span></span><span class="line"><span class="cl"><span class="sd"> DisableDNS=false </span></span></span><span class="line"><span class="cl"><span class="sd"> Internal=false</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Add the wordpress container.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/systemd/wordpress.container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">local</span><span class="p">:</span><span class="w"> </span><span class="l">quadlets/wordpress.container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="m">0644</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Add the MariaDB container.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/systemd/mariadb.container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">local</span><span class="p">:</span><span class="w"> </span><span class="l">quadlets/mariadb.container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="m">0644</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Add the caddy container.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/home/core/.config/containers/systemd/caddy.container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">local</span><span class="p">:</span><span class="w"> </span><span class="l">quadlets/caddy.container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="m">0644</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">core</span><span class="w"> </span></span></span></code></pre></div><p>The <a href="https://github.com/major/quadlets-wordpress/blob/main/caddy/Caddyfile" target="_blank" rel="noreferrer">Caddyfile</a> is also in the repository and will be deployed by the butane configuration shown above.</p> <p>We can go through each quadlet in detail. First up is MariaDB. We tell systemd that the wordpress container will want to have this one started first.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span> </span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">MariaDB Quadlet</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Container]</span> </span></span><span class="line"><span class="cl"><span class="na">Image</span><span class="o">=</span><span class="s">docker.io/library/mariadb:11</span> </span></span><span class="line"><span class="cl"><span class="na">ContainerName</span><span class="o">=</span><span class="s">mariadb</span> </span></span><span class="line"><span class="cl"><span class="na">AutoUpdate</span><span class="o">=</span><span class="s">registry</span> </span></span><span class="line"><span class="cl"><span class="na">EnvironmentFile</span><span class="o">=</span><span class="s">/home/core/.config/containers/containers-environment</span> </span></span><span class="line"><span class="cl"><span class="na">Volume</span><span class="o">=</span><span class="s">mariadb.volume:/var/lib/mysql</span> </span></span><span class="line"><span class="cl"><span class="na">Network</span><span class="o">=</span><span class="s">wordpress.network</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Service]</span> </span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">always</span> </span></span><span class="line"><span class="cl"><span class="na">TimeoutStartSec</span><span class="o">=</span><span class="s">900</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Install]</span> </span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">wordpress.service multi-user.target default.target</span> </span></span></code></pre></div><p>The wordpress quadlet is much the same as the MariaDB one, but we tell systemd that caddy will want wordpress started first.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span> </span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Wordpress Quadlet</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Container]</span> </span></span><span class="line"><span class="cl"><span class="na">Image</span><span class="o">=</span><span class="s">docker.io/library/wordpress:fpm</span> </span></span><span class="line"><span class="cl"><span class="na">ContainerName</span><span class="o">=</span><span class="s">wordpress</span> </span></span><span class="line"><span class="cl"><span class="na">AutoUpdate</span><span class="o">=</span><span class="s">registry</span> </span></span><span class="line"><span class="cl"><span class="na">EnvironmentFile</span><span class="o">=</span><span class="s">/home/core/.config/containers/containers-environment</span> </span></span><span class="line"><span class="cl"><span class="na">Volume</span><span class="o">=</span><span class="s">wordpress.volume:/var/www/html</span> </span></span><span class="line"><span class="cl"><span class="na">Network</span><span class="o">=</span><span class="s">wordpress.network</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Service]</span> </span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">always</span> </span></span><span class="line"><span class="cl"><span class="na">TimeoutStartSec</span><span class="o">=</span><span class="s">900</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Install]</span> </span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">caddy.service multi-user.target default.target</span> </span></span></code></pre></div><p>Finally, the caddy quadlet contains four volumes and some published ports. These ports will be published to the container host. Also, you&rsquo;ll note that the wordpress volume is mounted here, too. This is because caddy can serve static files <em>much faster</em> than wordpress can.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span> </span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">Caddy Quadlet</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Container]</span> </span></span><span class="line"><span class="cl"><span class="na">Image</span><span class="o">=</span><span class="s">docker.io/library/caddy:latest</span> </span></span><span class="line"><span class="cl"><span class="na">ContainerName</span><span class="o">=</span><span class="s">caddy</span> </span></span><span class="line"><span class="cl"><span class="na">AutoUpdate</span><span class="o">=</span><span class="s">registry</span> </span></span><span class="line"><span class="cl"><span class="na">EnvironmentFile</span><span class="o">=</span><span class="s">/home/core/.config/containers/containers-environment</span> </span></span><span class="line"><span class="cl"><span class="na">Volume</span><span class="o">=</span><span class="s">caddy-data.volume:/data</span> </span></span><span class="line"><span class="cl"><span class="na">Volume</span><span class="o">=</span><span class="s">caddy-config.volume:/config</span> </span></span><span class="line"><span class="cl"><span class="na">Volume</span><span class="o">=</span><span class="s">/home/core/.config/caddy/Caddyfile:/etc/caddy/Caddyfile:Z</span> </span></span><span class="line"><span class="cl"><span class="na">Volume</span><span class="o">=</span><span class="s">wordpress.volume:/var/www/html</span> </span></span><span class="line"><span class="cl"><span class="na">PublishPort</span><span class="o">=</span><span class="s">80:80</span> </span></span><span class="line"><span class="cl"><span class="na">PublishPort</span><span class="o">=</span><span class="s">443:443</span> </span></span><span class="line"><span class="cl"><span class="na">Network</span><span class="o">=</span><span class="s">wordpress.network</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Service]</span> </span></span><span class="line"><span class="cl"><span class="na">Restart</span><span class="o">=</span><span class="s">always</span> </span></span><span class="line"><span class="cl"><span class="na">TimeoutStartSec</span><span class="o">=</span><span class="s">900</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Install]</span> </span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target default.target</span> </span></span></code></pre></div><h1 id="launch-the-quadlets" class="relative group">Launch the quadlets <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#launch-the-quadlets" aria-label="Anchor">#</a></span></h1><p>There&rsquo;s a <a href="https://github.com/major/quadlets-wordpress/blob/main/launch-instance" target="_blank" rel="noreferrer">launch script</a> that ships this configuration to VULTR and launches a CoreOS instance:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="cp">#!/bin/bash </span></span></span><span class="line"><span class="cl"><span class="cp"></span><span class="c1"># This command starts up a CoreOS instance on Vultr using the vultr-cli</span> </span></span><span class="line"><span class="cl">vultr-cli instance create <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --os <span class="m">391</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --plan vhp-1c-1gb-amd <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --region dfw <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --notify <span class="nb">true</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --ipv6 <span class="nb">true</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> -u <span class="s2">&#34;</span><span class="k">$(</span>butane --files-dir . config.butane<span class="k">)</span><span class="s2">&#34;</span> <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> -l <span class="s2">&#34;coreos-</span><span class="k">$(</span>date <span class="s2">&#34;+%s&#34;</span><span class="k">)</span><span class="s2">&#34;</span> </span></span></code></pre></div><p>To launch an instance, get your <a href="https://my.vultr.com/settings/#settingsapi" target="_blank" rel="noreferrer">VULTR API key</a> first. Then install vultr-cli and butane:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo dnf -y install butane vultr-cli </span></span></code></pre></div><p>After launch, check to see what your containers are doing:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[core@vultr ~]$</span> podman ps </span></span><span class="line"><span class="cl"><span class="go">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></span></span><span class="line"><span class="cl"><span class="go">afa2d6501593 docker.io/library/caddy:latest caddy run --confi... 54 seconds ago Up 53 seconds 0.0.0.0:80-&gt;80/tcp, 0.0.0.0:443-&gt;443/tcp caddy </span></span></span><span class="line"><span class="cl"><span class="go">460426f39e6c docker.io/library/mariadb:11 mariadbd 35 seconds ago Up 35 seconds mariadb </span></span></span><span class="line"><span class="cl"><span class="go">92ece6538d5a docker.io/library/wordpress:fpm php-fpm 28 seconds ago Up 29 seconds wordpress </span></span></span></code></pre></div><p>We should be able to talk to wordpress through caddy on port 80:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[core@vultr ~]$</span> curl -si http://localhost/wp-admin/install.php <span class="p">|</span> head -n <span class="m">25</span> </span></span><span class="line"><span class="cl"><span class="go">HTTP/1.1 200 OK </span></span></span><span class="line"><span class="cl"><span class="go">Cache-Control: no-cache, must-revalidate, max-age=0 </span></span></span><span class="line"><span class="cl"><span class="go">Content-Type: text/html; charset=utf-8 </span></span></span><span class="line"><span class="cl"><span class="go">Expires: Wed, 11 Jan 1984 05:00:00 GMT </span></span></span><span class="line"><span class="cl"><span class="go">Server: Caddy </span></span></span><span class="line"><span class="cl"><span class="go">X-Powered-By: PHP/8.0.30 </span></span></span><span class="line"><span class="cl"><span class="go">Date: Mon, 25 Sep 2023 21:43:40 GMT </span></span></span><span class="line"><span class="cl"><span class="go">Transfer-Encoding: chunked </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">&lt;!DOCTYPE html&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;html lang=&#34;en-US&#34; xml:lang=&#34;en-US&#34;&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;head&gt; </span></span></span><span class="line"><span class="cl"><span class="go"> &lt;meta name=&#34;viewport&#34; content=&#34;width=device-width&#34; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go"> &lt;meta http-equiv=&#34;Content-Type&#34; content=&#34;text/html; charset=utf-8&#34; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go"> &lt;meta name=&#34;robots&#34; content=&#34;noindex,nofollow&#34; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go"> &lt;title&gt;WordPress &amp;rsaquo; Installation&lt;/title&gt; </span></span></span><span class="line"><span class="cl"><span class="go"> &lt;link rel=&#39;stylesheet&#39; id=&#39;dashicons-css&#39; href=&#39;http://localhost/wp-includes/css/dashicons.min.css?ver=6.3.1&#39; type=&#39;text/css&#39; media=&#39;all&#39; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;link rel=&#39;stylesheet&#39; id=&#39;buttons-css&#39; href=&#39;http://localhost/wp-includes/css/buttons.min.css?ver=6.3.1&#39; type=&#39;text/css&#39; media=&#39;all&#39; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;link rel=&#39;stylesheet&#39; id=&#39;forms-css&#39; href=&#39;http://localhost/wp-admin/css/forms.min.css?ver=6.3.1&#39; type=&#39;text/css&#39; media=&#39;all&#39; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;link rel=&#39;stylesheet&#39; id=&#39;l10n-css&#39; href=&#39;http://localhost/wp-admin/css/l10n.min.css?ver=6.3.1&#39; type=&#39;text/css&#39; media=&#39;all&#39; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;link rel=&#39;stylesheet&#39; id=&#39;install-css&#39; href=&#39;http://localhost/wp-admin/css/install.min.css?ver=6.3.1&#39; type=&#39;text/css&#39; media=&#39;all&#39; /&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;/head&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;body class=&#34;wp-core-ui language-chooser&#34;&gt; </span></span></span><span class="line"><span class="cl"><span class="go">&lt;p id=&#34;logo&#34;&gt;WordPress&lt;/p&gt; </span></span></span></code></pre></div><p>Awesome! πŸŽ‰</p> <h1 id="managing-containers" class="relative group">Managing containers <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#managing-containers" aria-label="Anchor">#</a></span></h1><p>Containers will automatically update on a schedule and you can check the timer:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[core@vultr ~]$</span> systemctl status --user podman-auto-update.timer </span></span><span class="line"><span class="cl"><span class="go">● podman-auto-update.timer - Podman auto-update timer </span></span></span><span class="line"><span class="cl"><span class="go"> Loaded: loaded (/usr/lib/systemd/user/podman-auto-update.timer; enabled; preset: disabled) </span></span></span><span class="line"><span class="cl"><span class="go"> Active: active (waiting) since Mon 2023-09-25 21:41:31 UTC; 3min 14s ago </span></span></span><span class="line"><span class="cl"><span class="go"> Trigger: Tue 2023-09-26 00:04:46 UTC; 2h 20min left </span></span></span><span class="line"><span class="cl"><span class="go"> Triggers: ● podman-auto-update.service </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Sep 25 21:41:31 vultr.guest systemd[1786]: Started podman-auto-update.timer - Podman auto-update timer. </span></span></span></code></pre></div><p>Quadlets are just regular systemd units:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[core@vultr ~]$</span> systemctl list-units --user <span class="p">|</span> grep -i Quadlet </span></span><span class="line"><span class="cl"><span class="go"> caddy.service loaded active running Caddy Quadlet </span></span></span><span class="line"><span class="cl"><span class="go"> mariadb.service loaded active running MariaDB Quadlet </span></span></span><span class="line"><span class="cl"><span class="go"> wordpress.service loaded active running Wordpress Quadlet </span></span></span></code></pre></div><p>As an example, you can make changes to caddy&rsquo;s config file and restart it easily:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">[core@vultr ~]$</span> systemctl restart --user caddy </span></span><span class="line"><span class="cl"><span class="gp">[core@vultr ~]$</span> systemctl status --user caddy </span></span><span class="line"><span class="cl"><span class="go">● caddy.service - Caddy Quadlet </span></span></span><span class="line"><span class="cl"><span class="go"> Loaded: loaded (/var/home/core/.config/containers/systemd/caddy.container; generated) </span></span></span><span class="line"><span class="cl"><span class="go"> Drop-In: /usr/lib/systemd/user/service.d </span></span></span><span class="line"><span class="cl"><span class="go"> └─10-timeout-abort.conf </span></span></span><span class="line"><span class="cl"><span class="go"> Active: active (running) since Mon 2023-09-25 21:46:28 UTC; 5s ago </span></span></span><span class="line"><span class="cl"><span class="go"> Main PID: 2652 (conmon) </span></span></span><span class="line"><span class="cl"><span class="go"> Tasks: 18 (limit: 1023) </span></span></span><span class="line"><span class="cl"><span class="go"> Memory: 15.1M </span></span></span><span class="line"><span class="cl"><span class="go"> CPU: 207ms </span></span></span></code></pre></div><p>If you need to change a quadlet&rsquo;s configuration, just open up the configuration file in your favorite editor under <code>~/.config/containers/systemd</code>, reload systemd, and restart the container:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> vi ~/.config/containers/systemd/caddy.container </span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">--- make your edits and save the quadlet configuration --- </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="gp">$</span> systemctl daemon-reload --user </span></span><span class="line"><span class="cl"><span class="gp">$</span> systemctl restart --user caddy </span></span></code></pre></div><p>Enjoy!</p>Mounting the AWS Elastic File Store on Fedorahttps://major.io/p/aws-elastic-file-system-fedora/Wed, 13 Sep 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/aws-elastic-file-system-fedora/<hr> <p>I package a few things here and there in <a href="https://fedoraproject.org/" target="_blank" rel="noreferrer">Fedora</a> and one of my latest packages is <a href="https://src.fedoraproject.org/rpms/efs-utils" target="_blank" rel="noreferrer">efs-utils</a>. AWS offers a mount helper for their <a href="https://aws.amazon.com/efs/" target="_blank" rel="noreferrer">Elastic File System (EFS)</a> product on <a href="https://github.com/aws/efs-utils" target="_blank" rel="noreferrer">GitHub</a>.</p> <p>In this post, I&rsquo;ll explain how to:</p> <ol> <li>Launch a Fedora instance on AWS EC2</li> <li>Install <em>efs-utils</em> and launch the watchdog service</li> <li>Create an EFS volume in the AWS console</li> <li>Mount the EFS volume inside the Fedora instance</li> </ol> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path fill="currentColor" d="M368 128C368 172.4 342.6 211.5 304 234.4V256C304 273.7 289.7 288 272 288H176C158.3 288 144 273.7 144 256V234.4C105.4 211.5 80 172.4 80 128C80 57.31 144.5 0 224 0C303.5 0 368 57.31 368 128V128zM168 176C185.7 176 200 161.7 200 144C200 126.3 185.7 112 168 112C150.3 112 136 126.3 136 144C136 161.7 150.3 176 168 176zM280 112C262.3 112 248 126.3 248 144C248 161.7 262.3 176 280 176C297.7 176 312 161.7 312 144C312 126.3 297.7 112 280 112zM3.379 273.7C11.28 257.9 30.5 251.5 46.31 259.4L224 348.2L401.7 259.4C417.5 251.5 436.7 257.9 444.6 273.7C452.5 289.5 446.1 308.7 430.3 316.6L295.6 384L430.3 451.4C446.1 459.3 452.5 478.5 444.6 494.3C436.7 510.1 417.5 516.5 401.7 508.6L224 419.8L46.31 508.6C30.5 516.5 11.28 510.1 3.379 494.3C-4.525 478.5 1.882 459.3 17.69 451.4L152.4 384L17.69 316.6C1.882 308.7-4.525 289.5 3.379 273.7V273.7z"/></svg> </span> </span> <span class="dark:text-neutral-300"><strong>Always check the pricing for any cloud service before you use it!</strong> <a href="https://aws.amazon.com/efs/pricing/" target="_blank" rel="noreferrer">EFS pricing</a> is based on how much you store and how often you access it. Backups are also enabled by default and they add to the monthly charges.</span> </div> <p>Let&rsquo;s go! πŸš€</p> <h1 id="wait-what-is-efs" class="relative group">Wait, what is EFS? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#wait-what-is-efs" aria-label="Anchor">#</a></span></h1><p>When you launch a cloud instance (virtual machine) on most clouds, you have different storage options available to you:</p> <ul> <li> <p><strong>Block storage:</strong> You can add partitions to this storage, create filesystems, or even use LVM. It looks like someone plugged in a disk to your instance. You get full control over every single storage block on the volume. An example of this is <a href="https://aws.amazon.com/ebs/" target="_blank" rel="noreferrer">Elastic Block Storage (EBS)</a> on AWS.</p> </li> <li> <p><strong>Object storage:</strong> Although you can&rsquo;t mount object storage (typically) within your instance, you can read/write objects to this storage via an API. You can upload nearly any type of file you can imagine as an object and then download it later. Objects can also have little bits of metadata attached to them and some of the metadata include <em>prefixes</em> which give a folder-like experience. AWS <a href="https://aws.amazon.com/s3/" target="_blank" rel="noreferrer">S3</a> is a good example of this.</p> </li> <li> <p><strong>Shared filesystems:</strong> This storage shows up in the instance exactly as it sounds: you get a shared filesystem. If you&rsquo;re familiar with NFS or Samba (SMB), then you&rsquo;ve used shared filesystems already. They give you much better performance than object storage but offer less freedom than block storage. They&rsquo;re also great for sharing the same data between multiple instances.</p> </li> </ul> <p>Using EFS is almost like having someone else host a network accessible storage (NAS) device within your cloud deployment.</p> <h1 id="launching-fedora" class="relative group">Launching Fedora <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#launching-fedora" aria-label="Anchor">#</a></span></h1><p>Every image in AWS has an AMI ID attached to it and you need to know the ID for the image you want in your region. You can find these quickly for Fedora by visiting the <a href="https://fedoraproject.org/cloud/download/" target="_blank" rel="noreferrer">Fedora Cloud download page</a>. Look for <em>AWS</em> in the list, click the button on that row, and you&rsquo;ll see a list of Fedora AMI IDs. Click the rocket (πŸš€) for your preferred region and you&rsquo;re linked directly to launch that instance in AWS!</p> <p>I&rsquo;m clicking the <a href="https://console.aws.amazon.com/ec2/home?region=us-east-2#launchAmi=ami-00ef4597fd9806efc" target="_blank" rel="noreferrer">launch link for us-east-2 (Ohio)</a><sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. To finish quickly, I&rsquo;m choosing all of the default options and using a spot instance (look inside <em>Advanced details</em> at the bottom of the page).</p> <p>Wait for the instance to finish intializing and access it via ssh:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> ssh fedora@EXTERNAL_IP </span></span><span class="line"><span class="cl"><span class="gp">[fedora@ip-172-31-2-38 ~]$</span> cat /etc/fedora-release </span></span><span class="line"><span class="cl"><span class="go">Fedora release 38 (Thirty Eight) </span></span></span></code></pre></div><p>Success! πŸŽ‰</p> <h1 id="prepare-your-security-group" class="relative group">Prepare your security group <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#prepare-your-security-group" aria-label="Anchor">#</a></span></h1><p>Before leaving the EC2 console, you need to make a note of the security group that you used for this instance. That&rsquo;s because EFS uses security groups to guard access to volumes. Follow these steps to find it:</p> <ol> <li>Click <em>Instances</em> on the left side of the EC2 console.</li> <li>Click on the row showing the instance we just created.</li> <li>In the bottom half of the screen, click the <em>Security</em> tab.</li> <li>Look for <em>Security groups</em> in the security details and copy the security group ID for later.</li> </ol> <p>It should be in the format <code>sg-[a-f0-9]*</code>.</p> <p>If you click the security group name (after saving it), you&rsquo;ll see the inbound rules associated with that security group. By default, items in the same security group can&rsquo;t talk to each other. We need to allow that so our EFS mount will work later.</p> <p>Click <em>Edit inbound rules</em> and do the following:</p> <ol> <li>Click <em>Add rule</em>.</li> <li>Choose <em>All traffic</em> in the <em>Type</em> column. <em>(You can narrow this down further later.)</em></li> <li>In the source box, look for the security group you just created along with your EC2 instance. If you took the default during the EC2 launch process, it might be named <code>launch-wizard-[0-9]+</code>.</li> <li>Click <em>Save rules</em>.</li> </ol> <h1 id="installing-efs-utils" class="relative group">Installing efs-utils <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#installing-efs-utils" aria-label="Anchor">#</a></span></h1><p>Let&rsquo;s start by getting the <em>efs-utils</em> package onto our new Fedora system:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo dnf -qy install efs-utils </span></span><span class="line"><span class="cl"><span class="go">Installed: </span></span></span><span class="line"><span class="cl"><span class="go"> efs-utils-1.35.0-2.fc38.noarch </span></span></span></code></pre></div><p>The package includes some configuration, a watchdog, and a mount helper:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> rpm -ql efs-utils </span></span><span class="line"><span class="cl"><span class="go">/etc/amazon </span></span></span><span class="line"><span class="cl"><span class="go">/etc/amazon/efs </span></span></span><span class="line"><span class="cl"><span class="go">/etc/amazon/efs/efs-utils.conf </span></span></span><span class="line"><span class="cl"><span class="go">/etc/amazon/efs/efs-utils.crt </span></span></span><span class="line"><span class="cl"><span class="go">/usr/bin/amazon-efs-mount-watchdog </span></span></span><span class="line"><span class="cl"><span class="go">/usr/lib/systemd/system/amazon-efs-mount-watchdog.service </span></span></span><span class="line"><span class="cl"><span class="go">/usr/sbin/mount.efs </span></span></span><span class="line"><span class="cl"><span class="go">/usr/share/doc/efs-utils </span></span></span><span class="line"><span class="cl"><span class="go">/usr/share/doc/efs-utils/CONTRIBUTING.md </span></span></span><span class="line"><span class="cl"><span class="go">/usr/share/doc/efs-utils/README.md </span></span></span><span class="line"><span class="cl"><span class="go">/usr/share/licenses/efs-utils </span></span></span><span class="line"><span class="cl"><span class="go">/usr/share/licenses/efs-utils/LICENSE </span></span></span><span class="line"><span class="cl"><span class="go">/usr/share/man/man8/mount.efs.8.gz </span></span></span><span class="line"><span class="cl"><span class="go">/var/log/amazon/efs </span></span></span></code></pre></div><p>Let&rsquo;s get the watchdog running so we have that ready later. The watchdog helps to build and tear down the encrypted connection when you mount and unmount an EFS volume:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo systemctl <span class="nb">enable</span> --now amazon-efs-mount-watchdog.service </span></span><span class="line"><span class="cl"><span class="go">Created symlink /etc/systemd/system/multi-user.target.wants/amazon-efs-mount-watchdog.service β†’ /usr/lib/systemd/system/amazon-efs-mount-watchdog.service. </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="gp">$</span> systemctl status amazon-efs-mount-watchdog.service </span></span><span class="line"><span class="cl"><span class="go">● amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog </span></span></span><span class="line"><span class="cl"><span class="go"> Loaded: loaded (/usr/lib/systemd/system/amazon-efs-mount-watchdog.service; enabled; preset: disabled) </span></span></span><span class="line"><span class="cl"><span class="go"> Drop-In: /usr/lib/systemd/system/service.d </span></span></span><span class="line"><span class="cl"><span class="go"> └─10-timeout-abort.conf </span></span></span><span class="line"><span class="cl"><span class="go"> Active: active (running) since Wed 2023-09-13 18:43:46 UTC; 5s ago </span></span></span><span class="line"><span class="cl"><span class="go"> Main PID: 1258 (amazon-efs-moun) </span></span></span><span class="line"><span class="cl"><span class="go"> Tasks: 1 (limit: 4385) </span></span></span><span class="line"><span class="cl"><span class="go"> Memory: 13.3M </span></span></span><span class="line"><span class="cl"><span class="go"> CPU: 76ms </span></span></span><span class="line"><span class="cl"><span class="go"> CGroup: /system.slice/amazon-efs-mount-watchdog.service </span></span></span><span class="line"><span class="cl"><span class="go"> └─1258 /usr/bin/python3 /usr/bin/amazon-efs-mount-watchdog </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Sep 13 18:43:46 ip-172-31-2-38.us-east-2.compute.internal systemd[1]: Started amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog. </span></span></span></code></pre></div><h1 id="setting-up-an-efs-volume" class="relative group">Setting up an EFS volume <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#setting-up-an-efs-volume" aria-label="Anchor">#</a></span></h1><p>Start by going over to the <a href="https://us-east-2.console.aws.amazon.com/efs/home?region=us-east-2#" target="_blank" rel="noreferrer">EFS console</a> and do the following:</p> <ol> <li> <p>Click <em>File systems</em> in the left navigation bar</p> </li> <li> <p>Click the orange <em>Create file system</em> button at the top right</p> <p>A modal appears with a box for the volume name and a VPC selection. Select an easy to remember name (I&rsquo;m using <em>testing-efs-for-blog-post</em>) and select a VPC. If you&rsquo;re not sure what a VPC is or which one to use, use the default VPC since that&rsquo;s likely where your instance landed as well.</p> </li> <li> <p>Click <em>Create</em>.</p> </li> </ol> <p>There&rsquo;s a delay while the filesystem initializes and you should see the filesystem show <em>Available</em> with a green check mark after about 30 seconds. Click on the filesystem you just created from the list and you&rsquo;ll see the details page for the filesystem.</p> <h1 id="security-setup" class="relative group">Security setup <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#security-setup" aria-label="Anchor">#</a></span></h1><p>EFS volumes come online with the default security group attached and that&rsquo;s not helpful. From the EFS filesystem details page, click the <em>Network</em> tab and then click <em>Manage</em>.</p> <p>For each availability zone, go to the <em>Security groups</em> column and add the security group that your instance came up with in the first step. In my case, I accepted the defaults from EC2 and ended up with a <em>launch-wizard-1</em> security group. Remove the <em>default</em> security group from each. Click <em>Save</em>.</p> <h1 id="mounting-time" class="relative group">Mounting time <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#mounting-time" aria-label="Anchor">#</a></span></h1><p>You should still be on the filesystem details page from the previous step. Click <em>Attach</em> at the top right and a modal will appear with mount instructions. The first option should use the EFS mount helper!</p> <p>For me, it looks like <code>sudo mount -t efs -o tls fs-0baabc62763375bb1:/ efs</code></p> <p>Go back to your Fedora instance, create a mount point, and create the volume:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo mkdir /mnt/efs </span></span><span class="line"><span class="cl"><span class="gp">$</span> sudo mount -t efs -o tls fs-0baabc62763375bb1:/ /mnt/efs </span></span><span class="line"><span class="cl"><span class="gp">$</span> df -hT <span class="p">|</span> grep efs </span></span><span class="line"><span class="cl"><span class="go">127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /mnt/efs </span></span></span></code></pre></div><p>We did it! πŸŽ‰</p> <p>We see <code>127.0.0.1</code> here because efs-utils uses stunnel to handle the encryption between your instance and the EFS storage system.</p> <p>The disk was mounted by root, so we can add a <code>-o user=fedora</code> to give our Fedora user permissions to write files:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> umount /mnt/efs </span></span><span class="line"><span class="cl"><span class="gp">$</span> sudo mount -t efs -o <span class="nv">user</span><span class="o">=</span>fedora,tls fs-0baabc62763375bb1:/ /mnt/efs </span></span><span class="line"><span class="cl"><span class="gp">$</span> touch /mnt/efs/test2.txt </span></span><span class="line"><span class="cl"><span class="gp">$</span> stat /mnt/efs/test2.txt </span></span><span class="line"><span class="cl"><span class="go"> File: /mnt/efs/test2.txt </span></span></span><span class="line"><span class="cl"><span class="go"> Size: 0 Blocks: 8 IO Block: 1048576 regular empty file </span></span></span><span class="line"><span class="cl"><span class="go">Device: 0,54 Inode: 17657675890899444015 Links: 1 </span></span></span><span class="line"><span class="cl"><span class="go">Access: (0644/-rw-r--r--) Uid: ( 1000/ fedora) Gid: ( 1000/ fedora) </span></span></span><span class="line"><span class="cl"><span class="go">Context: system_u:object_r:nfs_t:s0 </span></span></span><span class="line"><span class="cl"><span class="go">Access: 2023-09-13 19:14:23.308000000 +0000 </span></span></span><span class="line"><span class="cl"><span class="go">Modify: 2023-09-13 19:14:23.308000000 +0000 </span></span></span><span class="line"><span class="cl"><span class="go">Change: 2023-09-13 19:14:23.308000000 +0000 </span></span></span><span class="line"><span class="cl"><span class="go"> Birth: - </span></span></span></code></pre></div><p>Also, <em>efs-utils</em> uses encrypted communication by default, which is great. There may be some situations where you don&rsquo;t need encrypted communications or you don&rsquo;t want the overhead. In that case, drop the <code>-o tls</code> option from the mount command and you&rsquo;ll mount the volume unencrypted.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo umount /mnt/efs </span></span><span class="line"><span class="cl"><span class="gp">$</span> sudo mount -t efs -o <span class="nv">user</span><span class="o">=</span>fedora fs-0baabc62763375bb1:/ /mnt/efs </span></span><span class="line"><span class="cl"><span class="gp">$</span> df -hT <span class="p">|</span> grep efs </span></span><span class="line"><span class="cl"><span class="go">fs-0baabc62763375bb1.efs.us-east-2.amazonaws.com:/ nfs4 8.0E 0 8.0E 0% /mnt/efs </span></span></span></code></pre></div><h1 id="extra-credit" class="relative group">Extra credit <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#extra-credit" aria-label="Anchor">#</a></span></h1><p>You can get fancy with <a href="https://docs.aws.amazon.com/efs/latest/ug/create-access-point.html" target="_blank" rel="noreferrer">access points</a> that allow you to carve up your EFS storage and only let certain instances mount certain parts of the filesystem. So instance A might only be able to mount <code>/files/hr</code> while instance B can only mount <code>/documents</code>.</p> <p>It would also be a good idea to take an inventory of your security groups and ensure the least amount of instances can reach your EFS volume as possible. Much of the work I did in this post was just for testing. A good plan might be to make a security group for your EFS volume and only allow inbound traffic from security groups which should access it. That would allow you to gather up all of your instances into different security groups and limit access.</p> <p>Also, be aware of the <a href="https://aws.amazon.com/efs/pricing/" target="_blank" rel="noreferrer">EFS pricing</a>! πŸ’Έ</p> <p>You are billed not only for how much storage you use, but also on requests. Different requests are priced differently depending on access frequency. Backups are also <strong>enabled by default</strong> at $0.05/GB-month!</p> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>Why Ohio? I&rsquo;m mainly doing it to irritate <a href="https://www.lastweekinaws.com/" target="_blank" rel="noreferrer">Corey Quinn</a>. 🀭 Any region you prefer should be fine.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>Car buying guidehttps://major.io/p/car-buying-guide/Mon, 04 Sep 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/car-buying-guide/<hr> <p>I love optimizing nearly everything in my life. Sometimes it means saving money. Other times it means squeezing every bit of performance out of a server.</p> <p>But let&rsquo;s try optimizing something I&rsquo;ve never done on this blog before: <strong>Buying a car.</strong></p> <p>Although most of this information applies best to people in the USA, there are several things I&rsquo;ve learned over the years that might benefit people in other places. Car purchases are often the second largest purchase for most Americans after buying a house. Why not optimize it as much as possible?</p> <p>My family is in the market for new vehicle and I&rsquo;ve immersed myself in learning more about the whole process. There are plenty of legal issues in play that many buyers don&rsquo;t know about and there are tons of ways to walk into a dealership fully prepared.</p> <p>Without further ado, I&rsquo;ll share what I&rsquo;ve learned with you!</p> <h1 id="shopping" class="relative group">Shopping <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#shopping" aria-label="Anchor">#</a></span></h1><p>Shopping and buying are two different things. <strong>You must keep them separate.</strong> If you&rsquo;re not sure on the exact model of car you want, go shopping and don&rsquo;t commit to buy anything.</p> <p>If you have an idea of a particular car you want, go to Google and search for 5-10 competitors to that car. It&rsquo;s as easy as searching &ldquo;top competitors Camry&rdquo; to find other cars that compete with a Toyota Camry. Go to those dealers and take a good look at the cars to see if they have the features you want. Test drive each and see if you still like them.</p> <p>The really important thing here is to separate <strong>shopping</strong> from <strong>buying</strong>.</p> <h1 id="search-for-prices" class="relative group">Search for prices <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#search-for-prices" aria-label="Anchor">#</a></span></h1><p>Once you narrow your search down a bit, start to compare prices between dealers for different trim levels. I love using <a href="https://caredge.com/" target="_blank" rel="noreferrer">CarEdge</a> for this since you can examine dealer inventory across the country and see how dealer prices stack up against each other.</p> <p>Also check how long the vehicle has been sitting on the lot. CarEdge offers this data and it can give you an idea of how desperate a dealer is to get rid of a particular vehicle. If the total supply of a model is really high and the vehicle has been sitting on the lot longer than 90 days, you have the ability to negotiate for that car. Cars with a very low supply or cars that just arrived to the lot will likely be priced higher and dealers are less likely to budge on price.</p> <p>For used cars, this is even more important since you don&rsquo;t have an MSRP to work with as you do for new cars. Comparing prices for used cars is critical.</p> <p>Keep in mind that some cars are in higher demand in some areas than others, too. You might be able to drive a few hours and save quite a bit. A friend of mine drove from Texas to Colorado to buy a car and saved $3,500.</p> <p><strong>Watch out for arbitrary markups!</strong> All dealers in the US are required to display a Monroney label and this shows the manufacturer&rsquo;s suggested retail price. Dealers might add an addendum sticker somewhere else that show accessories they added. Sometimes these stickers show arbitrary markups from the dealer.</p> <p>Since COVID, many dealers have been stacking massive fees on top of car purchases. Some of these fees exceed $10,000, $25,000 or more! <strong>This should be a massive red flag from the start of a negotiation with any dealer.</strong> If they aren&rsquo;t willing to budge from their arbitrary markups, look for another dealer. 🚩</p> <h1 id="you-found-the-car-you-want" class="relative group">You found the car you want <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#you-found-the-car-you-want" aria-label="Anchor">#</a></span></h1><p>Awesome! πŸ₯³</p> <p>Whether it&rsquo;s new or used, take the VIN number and dig up information about the car. You can learn a lot from just a quick Google search from the VIN!</p> <p>Other than that, CarEdge has some good tools for digging up data on individual cars without too much expense. CarFax is the gold standard used and it offers some fairly inexpensive options.</p> <p>Some might be saying: <em>&ldquo;Isn&rsquo;t this a waste for a brand new car?&rdquo;</em> <strong>No, it&rsquo;s not.</strong></p> <p>I was once buying a pickup truck that was advertised as one model year, but the truck was actually a year older. The dealer even switched the paper tags attached to the keys so I wouldn&rsquo;t notice. I didn&rsquo;t notice this until months later when I found it buried in my paperwork. 😑</p> <p>There have also been situations where <strong>stolen cars</strong> are sitting on dealer lots. 😱</p> <p>Remember, this is a big investment. Spending $20 on a CarFax report as you purchase a $30,000 car should be worth it.</p> <h1 id="financing" class="relative group">Financing <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#financing" aria-label="Anchor">#</a></span></h1><p>Take care of your financing <strong>before visiting a dealer.</strong> Local credit unions are often great for this but they&rsquo;ve been under a squeeze lately with an increase in reposessions. Work with the credit union to get a good rate for the type of car you want.</p> <p>Some people have had good luck financing through a dealership. However, getting financing set up ahead of time gives you the upper hand with financing negotiations. For example, when you sit down with in the finance office, you could say: <em>&ldquo;I have 5.9% through my local credit union. If you can beat that, I&rsquo;ll finance with you.&rdquo;</em></p> <p>If you choose to go through financing with the dealer, here&rsquo;s what I recommend:</p> <ol> <li><strong>Do not allow the dealer to run your credit report until the last minute.</strong> If you let them run it, they know that you&rsquo;ll get a hard hit on your credit report and you&rsquo;re less likely to look at other dealers. You want to leave your options open. Don&rsquo;t let them run your credit until you are <strong>100% sure you&rsquo;re ready to buy from them</strong>.</li> <li>Ask to see the &ldquo;call sheet&rdquo; and see what the bank offered the dealer for financing (the &ldquo;buy rate&rdquo;). It&rsquo;s very likely that the dealer gets one rate from the bank and then marks it up for you. For example, the dealer might say <em>&ldquo;Oh, your rate is 8%.&rdquo;</em> Then you ask for the call sheet or the buy rate and see that they got 5.9% from the bank. <strong>You&rsquo;re getting a huge markup.</strong> That&rsquo;s another negotiation point.</li> <li>If the dealer says they can lower your interest rate if you buy an add-on from them, such as extended warranty, <strong>STOP IMMEDIATELY.</strong> 🚨 This is called &ldquo;tied selling&rdquo; and is almost always <a href="https://www.ftc.gov/advice-guidance/competition-guidance/guide-antitrust-laws/single-firm-conduct/tying-sale-two-products" target="_blank" rel="noreferrer">illegal in the USA</a>. This is a good moment to stop and re-evaluate the whole transaction.</li> <li>You are not required to buy any of the add-ons that the finance manager offers you. Extended warranties, dent and ding protection, and tire/wheel protection are common items. There&rsquo;s no requirement to purchase these. If you do decide to purchase one, be sure they can show you <strong>the actual amount that it will cost you.</strong> Don&rsquo;t let them tell you how much it adds to your monthly payment. That allows them to hide the cost of certain add-ons.</li> <li>You are required to pay reasonable fees for tax, title, and license. Sometimes documentation fees are rolled up into this, too. Every state is a little different on how much these cost, but you can Google &ldquo;tax title and license&rdquo; for your state and get a good estimation.</li> <li><strong>Demand to see the &ldquo;out the door price.&rdquo;</strong> If a dealer asks how much you can afford per month, <strong>don&rsquo;t answer.</strong> You are interested in the full price of the car plus fees. Most states require that this comes out on a single sheet of paper with each expense clearly labeled with what you will owe for the car. <strong>If you tell a dealer &ldquo;I can&rsquo;t affort more than $500 per month&rdquo;, then they will tinker with various parts of the deal to ensure you pay more in the long term without exceeding $500 per month.</strong> If the dealer won&rsquo;t budge, keep asking them &ldquo;If I was getting you a cashier&rsquo;s check today, how much needs to be on the check?&rdquo; They will eventually get the idea.</li> </ol> <p>Dealers make the most money by far not on the lot itself, but in the finance office. Don&rsquo;t get swindled there.</p> <h1 id="get-copies-and-read-them" class="relative group">Get copies and read them <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#get-copies-and-read-them" aria-label="Anchor">#</a></span></h1><p>You are entitled to a copy of everything that shows up in the finance office. <strong>Don&rsquo;t get up from the chair until you have a copy of everything and you&rsquo;ve examined each page.</strong></p> <p>Dealers commonly adjust numbers or conveniently leave out &ldquo;We Owe&rdquo; sheets (see the next section) in the finance office. Sometimes it&rsquo;s an honest mistake. Sometimes it&rsquo;s not.</p> <h1 id="the-dreaded-we-owe" class="relative group">The dreaded &ldquo;We Owe&rdquo; <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#the-dreaded-we-owe" aria-label="Anchor">#</a></span></h1><p>If a dealer doesn&rsquo;t have something in stock that they promised you, such as an accessory or add-on, ensure it lands on the &ldquo;We Owe&rdquo; sheet in your paperwork.</p> <p>There should be page somewhere in your sale paperwork that shows anything that the dealer owes you. In many states, this sheet <em>must</em> be in your paperwork even if it&rsquo;s blank or zeroed out!</p> <p>If a dealer promised you something and it didn&rsquo;t land on the &ldquo;We Owe&rdquo; sheet, stop immediately and ask for that to be corrected right then.</p> <h1 id="trade-ins" class="relative group">Trade-ins <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#trade-ins" aria-label="Anchor">#</a></span></h1><p>Any time you trade in a vehicle, <strong>make it a different transaction.</strong> Allowing a dealer to add your trade-in with the current deal for purchase allows them to hide money for themselves in the deal.</p> <p>First, get multiple offers from various sites that buy cars all day long. I recommend Carvana, Driveway, and Vroom. They will give you an immediate estimate online with a little bit of information. The CarEdge site I mentioned earlier also has some tools that allow you to look up your car in the Black Book, which is what dealers use to appraise cars.</p> <p>Next, when you go to the dealer and they ask if you&rsquo;re trading in a car, tell them you haven&rsquo;t decided yet. Your best bet is to act as if you want them to talk you into it. Either way, <strong>keep the trade-in as a separate transaction.</strong></p> <p>When it comes time to talk about your trade-in, you should already have an out the door price on your car purchase. <em>Scroll up if you&rsquo;re not sure about this.</em> πŸ˜‰</p> <p>Let the dealer know you have some other offers on your trade and let them know you&rsquo;ll do the trade there if they can beat the offers. If they offer to beat the other deals, that&rsquo;s awesome! That&rsquo;s less work for you!</p> <p>If they can&rsquo;t, don&rsquo;t worry. You can handle the trade-in separately with those other companies.</p> <h1 id="buying-online" class="relative group">Buying online <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#buying-online" aria-label="Anchor">#</a></span></h1><p>Finally, you can buy a car almost entirely online these days. Carvana, Driveway, and Vroom offer a fully online experience, but traditional dealers often have salespeople focused on internet deals.</p> <p>This is a good way to sort out dealers who want to work with you and those that don&rsquo;t. Let dealers know that:</p> <ol> <li>You know what vehicle you want.</li> <li>You&rsquo;re comparing the offers from multiple dealers.</li> <li>You&rsquo;re looking to purchase within the next few weeks.</li> <li>You want an out the door price on the vehicle so you know how much to get on the cashier&rsquo;s check.</li> </ol> <p>If they&rsquo;re willing to deal with you via email or phone, that&rsquo;s great! If not, there&rsquo;s plenty of other dealers out there.</p> <h1 id="further-learning" class="relative group">Further learning <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#further-learning" aria-label="Anchor">#</a></span></h1><p>I&rsquo;ve learned a lot from several YouTube channels over the years. Here are my favorites:</p> <ul> <li><a href="https://www.youtube.com/@carquestionsanswered" target="_blank" rel="noreferrer">Car Questions Answered</a>: Brandon runs a small used car dealership in North Carolina and gives lots of insights on what is happening the used and new car markets. He has lots of good advice on when to buy a car and which cars are the ones to avoid at certain times. If you ever wanted to go behind the scenes to see how a small used car dealership works, his channel is great.</li> <li><a href="https://www.youtube.com/@DeshoneTheAutoAdvisor" target="_blank" rel="noreferrer">Deshone The Auto Advisor</a>: Deshone has lots of helpful short videos. He does offer a membership program that comes with a fee, but his short videos cover a ton of car buying and leasing recommendations. If you&rsquo;re interested in leasing a car, be sure to watch some of his leasing videos.</li> <li><a href="https://www.youtube.com/@CarEdge" target="_blank" rel="noreferrer">CarEdge</a>: CarEdge offers tons of services to help with car buying including 1:1 coaching once you have a deal sheet from a dealer. However, they also have tons of videos that explain how to buy a car effectively. They do role playing for finance managers and salespeople that highlight certain areas where buyers usually lose. Their role play videos are highly recommended.</li> <li><a href="https://www.youtube.com/@LuckyLopez777" target="_blank" rel="noreferrer">Lucky Lopez</a>: Lucky has tons of insight from a dealer perspective and he follows lots of trends around pricing, supply, and reposessions. Most of his content might be too much for car buyers, but it&rsquo;s good information to know.</li> <li><a href="https://www.youtube.com/channel/UCziwLfSSPLWc8bp02Pc5azg" target="_blank" rel="noreferrer">Chevy Dude</a>: The Chevy Dude used to work for multiple dealers but now it running his own. He shares lots of sales tricks and secrets that help you prepare for making your next deal on a car &ndash; new or used.</li> </ul> <h1 id="did-i-miss-something" class="relative group">Did I miss something? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#did-i-miss-something" aria-label="Anchor">#</a></span></h1><p>Let me know if I missed something and I&rsquo;ll come back and edit this post! Just contact me via any of the methods below in the author block. ️⬇️</p>Fixing a ghost database migration failurehttps://major.io/p/ghost-db-migration-failure/Thu, 31 Aug 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/ghost-db-migration-failure/<p>I love learning about the <em>behind the scenes</em> aspects of just about everything. I do <a href="https://w5wut.com" target="_blank" rel="noreferrer">ham radio</a>, I self-host lots of my personal infrastructure, and I&rsquo;ve been learning more about the math behind the stock market for the last year or two.</p> <p>That led me to start a blog on <a href="https://unsplash.com/photos/l6mLi-iKUW0" target="_blank" rel="noreferrer">Ghost</a> to share my findings with others. I started <a href="https://thetanerd.com" target="_blank" rel="noreferrer">Theta Nerd</a><sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> earlier this summer.</p> <p>My deployment looked great when I started! Everything was automatically updated with <a href="https://major.io/p/watchtower/">watchtower</a> and running with <a href="https://major.io/p/docker-compose-on-coreos/" target="_blank" rel="noreferrer">docker-compose on Fedora CoreOS</a>. <em>(Click these links to read the posts on both topics!)</em></p> <p>However, I woke up one morning to my monitoring going off and my site was down. 😱</p> <h1 id="why-is-the-site-down" class="relative group">Why is the site down? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#why-is-the-site-down" aria-label="Anchor">#</a></span></h1><p>Anyone who has worked in IT knows this sinking feeling. Something is down, you don&rsquo;t know why, and you suspect the worst possible scenarios.</p> <p>The instance hosting the blog was online and responsive, so I started digging into the logs with <code>docker-compose logs</code>. I suddenly found a wall of text in the logs for the Ghost container:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Adding members.email_disabled column </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Setting email_disabled to true for all members that have their email on the suppression list </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Setting nullable: stripe_products.product_id </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Adding table: donation_payment_events </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Rolling back: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column &#39;member_id&#39; and referenced column &#39;id&#39; in foreign key constraint &#39;donation_payment_events_member_id_foreign&#39; are incompatible.. </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Dropping table: donation_payment_events </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Dropping nullable: stripe_products.product_id with foreign keys disabled </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Setting email_disabled to false for all members </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Removing members.email_disabled column </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] INFO Rollback was successful. </span></span><span class="line"><span class="cl">[2023-08-03 11:10:16] ERROR alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column &#39;member_id&#39; and referenced column &#39;id&#39; in foreign key constraint &#39;donation_payment_events_member_id_foreign&#39; are incompatible. </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column &#39;member_id&#39; and referenced column &#39;id&#39; in foreign key constraint &#39;donation_payment_events_member_id_foreign&#39; are incompatible. </span></span><span class="line"><span class="cl">{&#34;config&#34;:{&#34;transaction&#34;:false},&#34;name&#34;:&#34;2023-07-27-11-47-49-create-donation-events.js&#34;} </span></span><span class="line"><span class="cl">&#34;Error occurred while executing the following migration: 2023-07-27-11-47-49-create-donation-events.js&#34; </span></span><span class="line"><span class="cl">Error ID: </span></span><span class="line"><span class="cl"> 300 </span></span><span class="line"><span class="cl">Error Code: </span></span><span class="line"><span class="cl"> ER_FK_INCOMPATIBLE_COLUMNS </span></span><span class="line"><span class="cl">---------------------------------------- </span></span><span class="line"><span class="cl">Error: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column &#39;member_id&#39; and referenced column &#39;id&#39; in foreign key constraint &#39;donation_payment_events_member_id_foreign&#39; are incompatible. </span></span><span class="line"><span class="cl"> at /var/lib/ghost/versions/5.57.2/node_modules/knex-migrator/lib/index.js:1032:19 </span></span><span class="line"><span class="cl"> at Packet.asError (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packets/packet.js:728:17) </span></span><span class="line"><span class="cl"> at Query.execute (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/commands/command.js:29:26) </span></span><span class="line"><span class="cl"> at Connection.handlePacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:478:34) </span></span><span class="line"><span class="cl"> at PacketParser.onPacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:97:12) </span></span><span class="line"><span class="cl"> at PacketParser.executeStart (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packet_parser.js:75:16) </span></span><span class="line"><span class="cl"> at Socket.&lt;anonymous&gt; (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:104:25) </span></span><span class="line"><span class="cl"> at Socket.emit (node:events:513:28) </span></span><span class="line"><span class="cl"> at addChunk (node:internal/streams/readable:315:12) </span></span><span class="line"><span class="cl"> at readableAddChunk (node:internal/streams/readable:289:9) </span></span><span class="line"><span class="cl"> at Socket.Readable.push (node:internal/streams/readable:228:10) </span></span><span class="line"><span class="cl"> at TCP.onStreamRead (node:internal/stream_base_commons:190:23) </span></span></code></pre></div><p>Ah, so a failed database migration in the upgrade to 5.57.2 is the culprit! πŸ‘</p> <p>I brought the site back online quickly by changing the container version for Ghost back to the previous version (5.55.2).</p> <h1 id="why-did-the-database-migration-fail" class="relative group">Why did the database migration fail? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#why-did-the-database-migration-fail" aria-label="Anchor">#</a></span></h1><p>The error message from above boils down to this:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Error: alter table `donation_payment_events` add constraint </span></span><span class="line"><span class="cl">`donation_payment_events_member_id_foreign` foreign key (`member_id`) </span></span><span class="line"><span class="cl">references `members` (`id`) on delete SET NULL - Referencing column </span></span><span class="line"><span class="cl">&#39;member_id&#39; and referenced column &#39;id&#39; in foreign key constraint </span></span><span class="line"><span class="cl">&#39;donation_payment_events_member_id_foreign&#39; are incompatible. </span></span></code></pre></div><p>Adjusting the <code>donation_payment_events.member_id</code> column to be a foreign key of <code>members.id</code> is failing because they are incompatible types. However, as I examined both tables, both were regular <code>varchar(24)</code> columns without anything special attached to them:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">mysql&gt; describe members; </span></span><span class="line"><span class="cl">+------------------------------+---------------+------+-----+---------+-------+ </span></span><span class="line"><span class="cl">| Field | Type | Null | Key | Default | Extra | </span></span><span class="line"><span class="cl">+------------------------------+---------------+------+-----+---------+-------+ </span></span><span class="line"><span class="cl">| id | varchar(24) | NO | PRI | NULL | | </span></span><span class="line"><span class="cl">| uuid | varchar(36) | YES | UNI | NULL | | </span></span><span class="line"><span class="cl">| email | varchar(191) | NO | UNI | NULL | | </span></span><span class="line"><span class="cl">| status | varchar(50) | NO | | free | | </span></span><span class="line"><span class="cl">| name | varchar(191) | YES | | NULL | | </span></span><span class="line"><span class="cl">| expertise | varchar(191) | YES | | NULL | | </span></span><span class="line"><span class="cl">| note | varchar(2000) | YES | | NULL | | </span></span><span class="line"><span class="cl">| geolocation | varchar(2000) | YES | | NULL | | </span></span><span class="line"><span class="cl">| enable_comment_notifications | tinyint(1) | NO | | 1 | | </span></span><span class="line"><span class="cl">| email_count | int unsigned | NO | | 0 | | </span></span><span class="line"><span class="cl">| email_opened_count | int unsigned | NO | | 0 | | </span></span><span class="line"><span class="cl">| email_open_rate | int unsigned | YES | MUL | NULL | | </span></span><span class="line"><span class="cl">| last_seen_at | datetime | YES | | NULL | | </span></span><span class="line"><span class="cl">| last_commented_at | datetime | YES | | NULL | | </span></span><span class="line"><span class="cl">| created_at | datetime | NO | | NULL | | </span></span><span class="line"><span class="cl">| created_by | varchar(24) | NO | | NULL | | </span></span><span class="line"><span class="cl">| updated_at | datetime | YES | | NULL | | </span></span><span class="line"><span class="cl">| updated_by | varchar(24) | YES | | NULL | | </span></span><span class="line"><span class="cl">+------------------------------+---------------+------+-----+---------+-------+ </span></span><span class="line"><span class="cl">18 rows in set (0.00 sec) </span></span></code></pre></div><h1 id="going-upstream" class="relative group">Going upstream <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#going-upstream" aria-label="Anchor">#</a></span></h1><p>I went to Ghost&rsquo;s GitHub repository and <a href="https://github.com/TryGhost/Ghost/issues/17584" target="_blank" rel="noreferrer">opened an issue</a> with as much data as I can find.</p> <p>One of the <a href="https://github.com/TryGhost/Ghost/issues/17584#issuecomment-1671134556" target="_blank" rel="noreferrer">first replies</a> mentioned something about database collations. Long story short, collations describe how databases handle sorting and comparing data for different languages. Comparing some languages to other languages can be particularly challenging and this can lead to problems.</p> <p>I made a switch from MariaDB to MySQL recently for the blog. Could that be related?</p> <h1 id="more-searching" class="relative group">More searching <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#more-searching" aria-label="Anchor">#</a></span></h1><p>I figured that I wasn&rsquo;t the first one to stumble into this problem, and sure enough &ndash; I wasn&rsquo;t! There&rsquo;s a <a href="https://dnsmichi.at/2022/06/01/ghost-v5-upgrade-with-mysql-8-collation-migration-in-docker-compose/" target="_blank" rel="noreferrer">great blog post</a> about a broken migration from MySQL 5 to 8 with Ghost.</p> <p>In short, it required several steps to fix it:</p> <ol> <li>Stop the Ghost container</li> <li>Back up the database first (always a good idea)</li> <li>Do a quick find/replace on the dumped database to change the collations</li> <li>Drop the <code>ghost</code> database from the database 😱</li> <li>Import the database back into MySQL</li> <li>Start Ghost again</li> </ol> <p>Dropping databases always makes me pause, but that&rsquo;s what backups are for! πŸ˜‰</p> <h1 id="how-i-fixed-it" class="relative group">How I fixed it <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#how-i-fixed-it" aria-label="Anchor">#</a></span></h1><p>In my case, my MySQL container is called <code>ghostmysql</code> and my Ghost database is <code>ghostdb</code>. Then I made a backup of the database using <code>mysqldump</code>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo docker-compose exec ghostmysql mysqldump \ </span></span></span><span class="line"><span class="cl"><span class="go"> -u root -psuper-secret-password ghostdb &gt; backup-ghost-db.sql </span></span></span></code></pre></div><p>Next, I copied the SQL file to another directory <em>just in case</em> I accidentally deleted this backup with an errant command.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">cp backup-ghost-db.sql ../ </span></span></span></code></pre></div><p>Then I made a copy of the SQL file in the current directory and ran the find and replace on that copy. This changes the collations from the wrong one, <code>utf8mb4_general_ci</code>, to the right one, <code>utf8mb4_0900_ai_ci</code><sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">cp backup-ghost-db.sql backup-ghost-db-new.sql </span></span></span><span class="line"><span class="cl"><span class="go">sed -i &#39;s/utf8mb4_general_ci/utf8mb4_0900_ai_ci/g&#39; \ </span></span></span><span class="line"><span class="cl"><span class="go"> backup-ghost-db-new.sql </span></span></span></code></pre></div><p>Now I have the collations right for importing the database back into MySQL. But first, I have to drop the existing database. <strong>This is a good time to double check your backups!</strong></p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo docker-compose exec ghostmysql mysql -u root \ </span></span></span><span class="line"><span class="cl"><span class="go"> -psuper-secret-password </span></span></span><span class="line"><span class="cl"><span class="go">mysql&gt; DROP DATABASE ghostdb; </span></span></span></code></pre></div><p>Now we can import the modified backup:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">cat backup-ghost-db-new.sql | sudo docker-compose exec -T \ </span></span></span><span class="line"><span class="cl"><span class="go"> ghostmysql mysql -u root -psuper-secret-password ghostdb </span></span></span></code></pre></div><p>Start all the containers:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo docker-compose up -d </span></span></span></code></pre></div><p>Ghost was back online with the older version and everything looked good! I updated my <code>docker-compose.yaml</code> back to use <code>latest</code> for the Ghost version and ran <code>sudo docker-compose up -d</code> once more.</p> <p>Within seconds, the new container image was in place and the container was running! Both migrations completed in seconds and the blog was back online with the newest version. πŸŽ‰</p> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>Theta is one of many <a href="https://en.wikipedia.org/wiki/Greeks_%28finance%29" target="_blank" rel="noreferrer">financial Greeks</a> that measure certain aspects of options contracts in the market. It&rsquo;s also a <a href="https://en.wikipedia.org/wiki/Theta" target="_blank" rel="noreferrer">letter in the Greek alphabet</a>.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:2"> <p>The default collation in MySQL 8 is <code>utf8mb4_0900_ai_ci</code>.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>Open source contributions: Just do ithttps://major.io/p/open-source-like-nike/Wed, 16 Aug 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/open-source-like-nike/<p>I had a great time on the <a href="https://www.youtube.com/watch?v=aA-pBYxUPgU" target="_blank" rel="noreferrer">Fedora Podcast</a> yesterday to talk about Fedora cloud! We talked about all kinds of Fedora-related topics, but a couple of questions came up around how to contribute, especially when there&rsquo;s not a lot of structure in place for a particular type of contributions. Here&rsquo;s the full video if you&rsquo;re interested:</p> <div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"> <iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen="allowfullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/aA-pBYxUPgU?autoplay=1&controls=1&end=0&loop=0&mute=1&start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video" ></iframe> </div> <p>That made me think about a post that deserves to be written: <strong>How do you get started with open source contributions in a new project?</strong> πŸ€”</p> <p>My answer is pretty simple: <strong>Just do it.</strong> πŸš€</p> <h1 id="just-do-what" class="relative group">Just do what? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#just-do-what" aria-label="Anchor">#</a></span></h1><p>In the late 1980&rsquo;s, one of Nike&rsquo;s ad agencies <a href="https://en.wikipedia.org/wiki/Just_Do_It" target="_blank" rel="noreferrer">came up with the phrase</a> as a way to push through uncertainty. I was pretty young when this campaign started, but the general idea was this:</p> <ul> <li>Anyone can achieve what they want</li> <li>Stop worrying about whether you can actually do something</li> <li>Try something new</li> <li>Just do it</li> </ul> <p>Simple, right?</p> <p>This works for open source contributions, too. I often have conversations with people inside and outside of work where they identify a problem or an improvement in an open source project. My customary response is <em>&ldquo;Let&rsquo;s go upstream and make this better!&rdquo;</em></p> <p>However, what I hear back most often is <em>&ldquo;I don&rsquo;t know how.&rdquo;</em> This is where the whole <strong>just do it</strong> part comes in.</p> <h1 id="i-found-a-bug" class="relative group">I found a bug <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#i-found-a-bug" aria-label="Anchor">#</a></span></h1><p>Nearly every open source project wants to know about bugs that users experience. Start by finding out the best way to communicate with the people working on a particular project.</p> <p>For projects on GitHub or GitLab, you can open up an issue and describe your problem. Some repositories have a template generated for bugs that ask you several important questions, so be sure to follow those templates. If there isn&rsquo;t a template, I usually follow this format:</p> <ul> <li>What happened that was unexpected?</li> <li>What were you doing right before that unexpected event happened?</li> <li>What did you expect to happen instead?</li> <li>What else is nearby in the environment that might have an impact? <ul> <li>For example, the versions of Python might be important for Python-based projects.</li> </ul> </li> <li>What log files or other diagnostic materials exist?</li> </ul> <p>What&rsquo;s the goal? We want to give maintainers enough information for a quick diagnosis in the best case. If it&rsquo;s not obvious, then they need enough information to try to reproduce it on their own machine for debugging.</p> <p>Maintainers might come back with additional questions about your environment or the events just before the bug occurred. Be sure to respond in a timely way while the information is top of mind for them.</p> <p><strong>Always remember that these maintainers are real people who are likely not being paid for their work.</strong> Assume the best of intentions (unless proven otherwise) and stay focused on the solution. There might always be the chance that the maintainers are not interested in your use case and might not be interested in solving it.</p> <p>That leads me to the next step.</p> <h1 id="i-found-a-bug-and-i-want-to-fix-it" class="relative group">I found a bug and I want to fix it <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#i-found-a-bug-and-i-want-to-fix-it" aria-label="Anchor">#</a></span></h1><p>Start by opening an issue or a bug report first (see the previous section).</p> <p>This ensures that maintainers get a full picture of the problem you&rsquo;re trying to solve. Also, I&rsquo;ve had maintainers immediately reply and tell me that it&rsquo;s a known issue already being solved in another issue. That could save you some work.</p> <p>If you have a patch that fixes the issue, go through the following steps before submitting the fix upstream:</p> <ul> <li>Ensure your fix references the issue or bug report that you opened</li> <li>Use a very clear first line in your commit message, such as <code>parser: Fix emoji handling in YAML</code> rather than <code>Fix YAML bug</code></li> <li>Include a very brief explanation of the bug you&rsquo;re fixing in the commit message</li> <li><strong>Extra credit:</strong> Add or update existing tests so they catch the bug you just found</li> <li><strong>Extra credit:</strong> Add or update the project documentation for your change if necessary</li> </ul> <p>These extra credit items often make it easier to review your patch. Maintainers love extra test coverage, too.</p> <p>Submit your change in a pull request or merge request and watch for updates. Be patient with replies from the maintainers, but be timely in your replies. Remember that your use case might be an edge case for the upstream project and you might need to explain your fix (or the original bug) in more detail.</p> <h1 id="i-want-to-improve-something" class="relative group">I want to improve something <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#i-want-to-improve-something" aria-label="Anchor">#</a></span></h1><p>Improving an open source project could involve several things, such as:</p> <ul> <li>Enhancing by adding a new feature</li> <li>Optimizing an existing feature</li> <li>Creating documentation</li> <li>Building integrations</li> </ul> <p>I strongly recommend opening an issue first with the project maintainers to explain your enhancement. These <em>Requests for Enhancements</em>, or RFEs, should include several things:</p> <ul> <li>Your use case that made you think of the enhancement in the first place</li> <li>What you plan to add, substract, or change</li> <li>How the changes might affect different users, especially as they upgrade from older versions</li> <li>How the changes might affect testing or release processes</li> <li>Any changes in dependencies required</li> </ul> <p>Before going down the road of enhancements, always bring up these ideas with the maintainers first. You want to ensure that your ideal changes are aligned with the future goals of the project. In addition, maintainers will want to better understand your use case.</p> <p>Remember that an enhancement almost always requires additional work from maintainers. Every new use case means more work to ensure the project still functions. That&rsquo;s why it&rsquo;s critical to share your use case and have a good plan for testing and documentation.</p> <h1 id="getting-involved" class="relative group">Getting involved <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#getting-involved" aria-label="Anchor">#</a></span></h1><p>Whenever I find an open source project that I&rsquo;d like to get involved with, I start looking around for several things:</p> <ul> <li><strong>What do they use for informal asynchronous chat?</strong> IRC? Matrix? Slack? Something else? I join the chat, introduce myself, and get an idea for how they interact. Some groups are very chatty and informal while others are much more formal and regimented.</li> <li><strong>Where do they have detailed discussions?</strong> Many projects have detailed discussions in their issues/bugs or in places like GitHub&rsquo;s discussions. Others use <a href="https://major.io/p/mailing-list-beef/">old school mailing lists</a>. Some groups have regular meetings where anyone can add agenda items for discussions. If I need to talk about something a bit more long form and I expect some back and forth on it, I look for this avenue.</li> <li><strong>What requirements exist for contributors?</strong> Some projects require that contributors sign a <a href="https://en.wikipedia.org/wiki/Contributor_License_Agreement" target="_blank" rel="noreferrer">CLA</a> or some other sort of agreement. Make sure that any CLAs you sign are approved by your employer (if applicable). You might need an account on a system that you don&rsquo;t have, so check for that as well.</li> </ul> <p>From there, I take the <strong>just do it</strong> mentality and go for it. The worst thing you&rsquo;ll be told is <em>&ldquo;No&rdquo;</em>. If that happens, take a step back, see if there&rsquo;s another way to approach it, and try again.</p> <p>Remember one thing most of all: <strong>avoid taking anything personally.</strong> All of us have our bad days and some people have personalities that might be totally incompatible with yours (and most people in general). 🀭</p>Add CloudFront CDN to a Ghost bloghttps://major.io/p/ghost-cloudfront-cdn/Mon, 03 Jul 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/ghost-cloudfront-cdn/<p>After I launched my new <a href="https://thetanerd.com/" target="_blank" rel="noreferrer">stock market blog</a> on a self-hosted <a href="https://ghost.org/" target="_blank" rel="noreferrer">Ghost</a>, I wrote up the <a href="https://major.io/p/deploy-ghost/">deployment process</a> in containers last week. Then I had a shower thought: <em>How do I put a CDN in front of that?</em></p> <p>This blog is back on an <a href="https://major.io/p/cloudfront-migration/">S3 + CloudFront deployment</a> at AWS and I figured CloudFront could work well for a self-hosted Ghost blog, too.</p> <p>There are <strong>tons</strong> of blog posts out there that have outdated processes or only show you how to do one piece of the CDN deployment for Ghost. I read most of them and cobbled together a working deployment. Read on to learn how to do this yourself!</p> <h1 id="why-add-a-cdn" class="relative group">Why add a CDN? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#why-add-a-cdn" aria-label="Anchor">#</a></span></h1><p>Content Delivery Networks (CDN) enhance websites by doing a combination of different things:</p> <ol> <li><strong>High throughput content delivery.</strong> CDNs have extremely well connected systems with plenty of bandwidth available. When your web traffic goes overboard or a popular person links to your site, CDNs allow you to continue serving content at very high rates.</li> <li><strong>Cached content.</strong> CDNs will pull content from your origin server (the one running your application) and cache that content for you. This means fewer requests to your origin server and less bandwidth consumed there.</li> <li><strong>Content closer to consumers.</strong> You might host your site in the eastern USA, but a CDN can cache your content around the world for faster access. Your website might normally be slow for someone in Tokyo, but a local CDN endpoint in Japan could serve that content immediately there.</li> <li><strong>Improved security.</strong> Many CDNs offer a web application firewall (WAF) that allows you to limit access to certain functions on your site. This could prevent or slow down certain types of attacks that could take your site offline.</li> </ol> <p>CDNs have trade-offs, though. <strong>They&rsquo;re complicated.</strong></p> <p>They often require lots of DNS changes. TLS certificates remain a challenge. Caching solves lots of problems but can create headaches in a flash. A misconfiguration at the CDN level can take down your site or prevent it from operating properly for longer periods of time.</p> <p>Careful planning helps a lot! <em>Measure twice, cut once.</em></p> <h1 id="aws-terminology" class="relative group">AWS terminology <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#aws-terminology" aria-label="Anchor">#</a></span></h1><p>The names of various AWS services often confuse me, but here&rsquo;s what we need for this project:</p> <ul> <li><strong>AWS Certificate Manager:</strong> handles TLS certificate issuance and renewal for the CDN distribution</li> <li><strong>AWS CloudFront:</strong> the actual CDN itself</li> </ul> <p>CloudFront has a concept of <em>distributions</em>, which is a single configuration of the CDN for a particular site. We will get to that in the CloudFront section. πŸ˜‰</p> <h1 id="certificates" class="relative group">Certificates <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#certificates" aria-label="Anchor">#</a></span></h1><p>First off, we need a certificate for TLS connections. Run over to the <a href="https://us-east-1.console.aws.amazon.com/acm/home?region=us-east-1#/certificates/list" target="_blank" rel="noreferrer">AWS Certificate Manager (ACM) console</a> for your preferred region and follow these steps:</p> <ol> <li>Click the orange <strong>Request</strong> button at the top right.</li> <li>Request a public certificate on the next page and click <strong>Next</strong>.</li> <li>Type in the domain for your certificate that your users will type to access your site. For example, <code>example.com</code> or <code>blog.example.com</code>.</li> <li>Click <strong>Request</strong></li> </ol> <p>You should be back to your certificate list. Refresh the page by clicking on the circle with the arrow at the top right. Click on the certificate for the domain name you just added.</p> <p>In the second detail block labeled <strong>Details</strong>, look for the CNAME name and value at the far right. You need to set both of these wherever you host your DNS records. If you use AWS Route 53 for DNS, there&rsquo;s a button you can click there to do it immediately. If you use another DNS provider, create a CNAME record with the exact text shown there.</p> <p>Once you create those DNS records, go back to the page with your certificate and wait for it to change from <em>Pending validation</em> to <em>Issued</em>. This normally takes 2-3 minutes for most DNS providers I use.</p> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300">Wait for this to turn green and say <em>Issued</em> before proceeding to the next step!</span> </div> <p>Now that you have a certificate, it&rsquo;s time to configure our CDN distribution.</p> <h1 id="cloudfront" class="relative group">CloudFront <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#cloudfront" aria-label="Anchor">#</a></span></h1><p>Now comes the fun, but complicated part. You have two DNS records to think about here:</p> <ul> <li>The CDN DNS record that users will type to access your site, such as <code>example.com</code>.</li> <li>The origin DNS record that the CDN will use to access your backend Ghost blog, such as <code>origin.example.com</code>.</li> </ul> <p>The <em>origin</em> record will be hidden away behind the CDN when we&rsquo;re done.</p> <h2 id="create-the-distribution" class="relative group">Create the distribution <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#create-the-distribution" aria-label="Anchor">#</a></span></h2><p>Go to the <a href="https://us-east-1.console.aws.amazon.com/cloudfront/v3/home?region=us-east-2#/distributions" target="_blank" rel="noreferrer">CloudFront console</a> in your preferred region and follow these steps:</p> <ol> <li>Click <strong>Create Distribution</strong> at the top right.</li> <li>Put your origin (hidden) domain in <em>Origin domain</em>, such as <code>origin.example.com</code>.</li> <li>Skip down to <strong>Name</strong> for the distribution such as <em>&ldquo;My Ghost Blog&rdquo;</em>. <em>(This is for your internal use only.)</em></li> <li><strong>Compress objects automatically:</strong> <code>Yes</code></li> <li><strong>Viewer protocol policy:</strong> <code>Redirect HTTP to HTTPS</code></li> <li><strong>Allowed HTTP methods:</strong> <code>GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE</code></li> <li><strong>Cache policy:</strong> <code>CachingOptimized</code></li> <li><strong>Origin request policy:</strong> <code>AllViewerExceptHostHeader</code></li> <li><strong>WAF</strong>: <code>Do not enable security protections</code> <em>(This costs extra and you can tweak this configuration later if needed.)</em></li> <li><strong>Alternate domain name (CNAME):</strong> Use the DNS name that your users will access, such as <code>example.com</code></li> <li><strong>Custom SSL certificate:</strong> Choose the certificate we created in the previous section</li> <li>Click <strong>Create distribution</strong></li> </ol> <p>This can take up to 10 minutes to deploy once you&rsquo;re finished. At this point, we have an aggressive caching policy that will cause problems when members attempt to sign in or manage their membership. It will also break the Ghost administrative area.</p> <p>Let&rsquo;s fix that next.</p> <h2 id="adjust-caching" class="relative group">Adjust caching <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#adjust-caching" aria-label="Anchor">#</a></span></h2><p>Find the CloudFront distribution we just created and click the <strong>Behaviors</strong> tab. We are going to make three different sets of behavior configurations to handle the dynamic pages.</p> <p>Click <strong>Create Behavior</strong> and do the following:</p> <ol> <li>Enter <code>/ghost*</code> as the path pattern.</li> <li>Choose the origin from the drop down that you specified when creating the distribution.</li> <li><strong>Compress objects automatically:</strong> <code>Yes</code></li> <li><strong>Viewer protocol policy:</strong> <code>Redirect HTTP to HTTPS</code></li> <li><strong>Allowed HTTP methods:</strong> <code>GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE</code></li> <li><strong>Cache policy:</strong> <code>CachingDisabled</code></li> <li><strong>Origin request policy:</strong> <code>AllViewer</code></li> <li>Click <strong>Save changes</strong></li> </ol> <p>That takes care of the administrative interface. Now let&rsquo;s fix the caching on the members page:</p> <ol> <li>Enter <code>/members*</code> as the path pattern.</li> <li>Choose the origin from the drop down that you specified when creating the distribution.</li> <li><strong>Compress objects automatically:</strong> <code>Yes</code></li> <li><strong>Viewer protocol policy:</strong> <code>Redirect HTTP to HTTPS</code></li> <li><strong>Allowed HTTP methods:</strong> <code>GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE</code></li> <li><strong>Cache policy:</strong> <code>CachingDisabled</code></li> <li><strong>Origin request policy:</strong> <code>AllViewer</code></li> <li>Click <strong>Save changes</strong></li> </ol> <p>With this configuration, we have caching for all content except for the administrative and member interfaces.</p> <h1 id="testing" class="relative group">Testing <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#testing" aria-label="Anchor">#</a></span></h1><p>There are a few different ways to test at this point, but I prefer to go with an old tried and true method: the <code>/etc/hosts</code> file. 😜</p> <p>CloudFront offers a domain name on <code>*.cloudfront.net</code> that you can use, but it&rsquo;s not quite the same. Cookies for the admin/member interface don&rsquo;t always work since they cross domains and sometimes you&rsquo;re redirected back to the original domain name which bypasses the CDN altogether.</p> <p>Go back to the list of distributions in your <a href="https://us-east-1.console.aws.amazon.com/cloudfront/v3/home?region=us-east-2#/distributions" target="_blank" rel="noreferrer">CloudFront console</a> in your preferred region. Click on the distribution you created earlier. At the top left, you&rsquo;ll see <strong>Distribution domain name</strong> with a domain underneath that contains <code>random_text.cloudfront.net</code>.</p> <p>Take that domain name and get an IPv4 address:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> dig +short A d2xznlk9a1h8zn.cloudfront.net </span></span><span class="line"><span class="cl"><span class="go">18.161.156.2 </span></span></span><span class="line"><span class="cl"><span class="go">18.161.156.18 </span></span></span><span class="line"><span class="cl"><span class="go">18.161.156.61 </span></span></span><span class="line"><span class="cl"><span class="go">18.161.156.9 </span></span></span></code></pre></div><p>Open <code>/etc/hosts</code> in your favorite editor (root access required) and use one of the IP addresses that correspond to your CDN endpoint. Add a line like this one (using your CDN domain and IPv4 address from the last step):</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plain" data-lang="plain"><span class="line"><span class="cl">18.161.156.2 example.com </span></span></code></pre></div><p>Access your site in a browser and verify that everything works. Be sure that you can access the administrative console under <code>example.com/ghost</code> and any member settings.</p> <p>️Remove the line in <code>/etc/hosts</code> now that we&rsquo;re finished with testing.</p> <h1 id="production" class="relative group">Production <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#production" aria-label="Anchor">#</a></span></h1><p>Our first step is to set up the origin.</p> <h2 id="origin-configuration" class="relative group">Origin configuration <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#origin-configuration" aria-label="Anchor">#</a></span></h2><p>Ensure your origin server has a proper DNS record so that CloudFront can access it on the backend. For example, <code>origin.example.com</code> must have a DNS record that points to your backend server running Ghost.</p> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300">Verify that the DNS record for your origin works before proceeding. πŸ’£</span> </div> <p>If you followed my <a href="https://major.io/p/deploy-ghost/">guide for deploying Ghost</a>, then you need to adjust your caddy configuration to answer requests to your origin URL. I updated my Caddyfile to contain both the origin and CDN hostnames:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-caddy" data-lang="caddy"><span class="line"><span class="cl"><span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">email</span> <span class="s">major@mhtx.net</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span><span class="line"><span class="cl"><span class="gh">thetanerd.com</span>, <span class="gh">origin.thetanerd.com</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">reverse_proxy</span> <span class="n">ghost</span><span class="p">:</span><span class="mi">2368</span> </span></span><span class="line"><span class="cl"> <span class="k">log</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">output</span> <span class="s">stderr</span> </span></span><span class="line"><span class="cl"> <span class="k">format</span> <span class="s">console</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="gh">www.thetanerd.com</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">redir</span> <span class="s">https://thetanerd.com</span><span class="se">{uri}</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span></code></pre></div><p>Restart caddy with <code>sudo docker-compose restart caddy</code>.</p> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300">Verify that caddy responds to requests <strong>to the origin hostname</strong> before going any further. It must respond properly with a valid SSL/TLS certificate! πŸ’£</span> </div> <h2 id="big-switch" class="relative group">Big switch <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#big-switch" aria-label="Anchor">#</a></span></h2><p>Now that our origin server is happy and responding, it&rsquo;s time to make the big switch. We&rsquo;re going to remove the record for the main CDN domain, such as <code>example.com</code> and replace it with a CNAME or ALIAS record to the CDN name in CloudFront. This is the name that ends in <code>cloudfront.net</code> that we used for testing earlier.</p> <p>The use of a CNAME or ALIAS record depends on your DNS host and the type of domain name you&rsquo;re using for the CDN.</p> <ul> <li>If you&rsquo;re using apex domain name (no subdomain) such as <code>example.com</code>, you will likely need to use an <code>ALIAS</code> record</li> <li>For domain names with a subdomain, such as <code>blog.example.com</code>, you will likely need to use a <code>CNAME</code> record</li> </ul> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300">Read your DNS host&rsquo;s documentation if you are unsure about ALIAS vs CNAME records! πŸ’£</span> </div> <p>Go your DNS registrar and follow these steps:</p> <ul> <li>Screenshot your existing DNS records or export them if possible (in case you need to revert).</li> <li>Remove the existing A/AAAA/CNAME/ALIAS record(s) for your main domain name, such as <code>example.com</code>.</li> <li>Immediately add a CNAME/ALIAS record from <code>example.com</code> to <code>random_text.cloudfront.net</code> that corresponds to your CloudFront distribution.</li> </ul> <p>Once that&rsquo;s done, I usually run <code>curl</code> in a terminal to watch for the changeover with <code>watch curl -si https://example.com</code>. When CloudFront is handling your traffic you&rsquo;ll see headers like these:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plain" data-lang="plain"><span class="line"><span class="cl">HTTP/2 200 </span></span><span class="line"><span class="cl">content-type: text/html; charset=utf-8 </span></span><span class="line"><span class="cl">cache-control: public, max-age=0 </span></span><span class="line"><span class="cl">date: Mon, 03 Jul 2023 19:44:55 GMT </span></span><span class="line"><span class="cl">server: Caddy </span></span><span class="line"><span class="cl">x-powered-by: Express </span></span><span class="line"><span class="cl">etag: W/&#34;19e7b-q5fZSjf8acC7o9lhdO5R+jOASfM&#34; </span></span><span class="line"><span class="cl">vary: Accept-Encoding </span></span><span class="line"><span class="cl">x-cache: Miss from cloudfront </span></span><span class="line"><span class="cl">via: 1.1 b2ba542a917451d9d85e07dba0cfd9a4.cloudfront.net (CloudFront) </span></span><span class="line"><span class="cl">x-amz-cf-pop: DFW57-P2 </span></span><span class="line"><span class="cl">x-amz-cf-id: Tpcjk886L0xAZzOjuUP-js_7-twE7ZGDZKlkmGHNTjW8hEs7oOWaLg== </span></span></code></pre></div><p>If it seems like it&rsquo;s taking a very long time to change over, use a tool like <a href="https://dnschecker.org/" target="_blank" rel="noreferrer">DNS Checker</a> to see how various DNS servers see your recent DNS change.</p> <h2 id="revert-if-needed" class="relative group">Revert (if needed) <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#revert-if-needed" aria-label="Anchor">#</a></span></h2><p>If something went horribly wrong, <strong>DON&rsquo;T PANIC</strong>. 😱</p> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300"><strong>DNS is like IT quicksand.</strong> Once you get stuck in a problem with DNS, any level of fighting just makes you more stuck. Take a deep breath first. 🫁</span> </div> <p>Go back to your DNS provide and remove the ALIAS/CNAME record for your CDN domain name, such as <code>example.com</code>. Add back in the original A/AAAA/ALIAS/CNAME records that were there previously. Be patient for traffic to shift back to your origin server.</p> <p>Review the changes you made and look for any errors.</p> <h1 id="configuring-ghost" class="relative group">Configuring Ghost <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#configuring-ghost" aria-label="Anchor">#</a></span></h1><p>Ghost is fairly easy to put behind a CDN, but it does have some additional caching configuration that you can change if needed. It provides hints to the CDN about what should and should not be cached and for how long. Refer to the <a href="https://ghost.org/docs/config/#caching" target="_blank" rel="noreferrer">Ghost docs</a> for details.</p> <p>I decided to cache requests to the Content API and to the frontend for 60 seconds as a test. My <code>docker-compose.yml</code> now looks like this:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ghost</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/library/ghost:5</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">ghost</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">always</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">https://thetanerd.com</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">caching__contentAPI__maxAge</span><span class="p">:</span><span class="w"> </span><span class="m">60</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">caching__frontend__maxAge</span><span class="p">:</span><span class="w"> </span><span class="m">60</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__client</span><span class="p">:</span><span class="w"> </span><span class="l">mysql</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__host</span><span class="p">:</span><span class="w"> </span><span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__user</span><span class="p">:</span><span class="w"> </span><span class="l">ghost</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__password</span><span class="p">:</span><span class="w"> </span><span class="l">...</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__database</span><span class="p">:</span><span class="w"> </span><span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ghost:/var/lib/ghost/content</span><span class="w"> </span></span></span></code></pre></div><p>Now if I access the main page of the site, I see cache hits in the headers:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-plain" data-lang="plain"><span class="line"><span class="cl">HTTP/2 200 </span></span><span class="line"><span class="cl">content-type: text/html; charset=utf-8 </span></span><span class="line"><span class="cl">cache-control: public, max-age=600 </span></span><span class="line"><span class="cl">date: Mon, 03 Jul 2023 19:54:39 GMT </span></span><span class="line"><span class="cl">etag: W/&#34;19e7b-5MKnFrme/sGk5DT2yvMkbgDsl+4&#34; </span></span><span class="line"><span class="cl">server: Caddy </span></span><span class="line"><span class="cl">x-powered-by: Express </span></span><span class="line"><span class="cl">vary: Accept-Encoding </span></span><span class="line"><span class="cl">x-cache: Hit from cloudfront </span></span><span class="line"><span class="cl">via: 1.1 308bae6dc9384ec8e0a82ba2d96014bc.cloudfront.net (CloudFront) </span></span><span class="line"><span class="cl">x-amz-cf-pop: DFW57-P2 </span></span><span class="line"><span class="cl">x-amz-cf-id: 0Dvoc_ST8-FK_TD4lEMQg6-uiDqhaUbYAqbylkiUP61eGcQsZSFEGg== </span></span><span class="line"><span class="cl">age: 7 </span></span></code></pre></div><p>The <code>x-cache</code> header shows a hit and the <code>age</code> header says it&rsquo;s been cached for 7 seconds.</p> <p>Enjoy your new CDN-accelerated Ghost blog! πŸ‡</p>Deploy a containerized Ghost blog πŸ‘»https://major.io/p/deploy-ghost/Tue, 27 Jun 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/deploy-ghost/<p>There&rsquo;s no shortage of options for starting a self-hosted blog. Wordpress might be chosen most often, but I stumbled upon <a href="https://ghost.org/" target="_blank" rel="noreferrer">Ghost</a> recently and their <a href="https://ghost.org/vs/wordpress/" target="_blank" rel="noreferrer">performance numbers</a> really got my attention.</p> <p>I prefer deploying most things in containers these days with <a href="https://fedoraproject.org/coreos/" target="_blank" rel="noreferrer">Fedora CoreOS</a>. Luckily, the Ghost stack doesn&rsquo;t demand a lot of infrastructure:</p> <ul> <li>Ghost itself</li> <li>MySQL 8+ <em>(I went with MariaDB 11.x)</em></li> <li>A web server out front</li> <li>TLS certificate</li> </ul> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300">Although I chose MariaDB for the database here, <strong>Ghost recommends MySQL</strong> and will throw a warning in the admin panel if you&rsquo;re using something else. I haven&rsquo;t had any issues so far, but <strong>you&rsquo;ve been warned</strong>. πŸ’£</span> </div> <p>I picked Caddy for the webserver since it&rsquo;s so small and the configuration is tremendously simple.</p> <h1 id="launch-coreos" class="relative group">Launch CoreOS <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#launch-coreos" aria-label="Anchor">#</a></span></h1><p>Fedora CoreOS offers lots of <a href="https://fedoraproject.org/coreos/download/?stream=stable" target="_blank" rel="noreferrer">cloud options</a> for launching it immediately. Many public clouds already have CoreOS images available, but I love Hetzner&rsquo;s US locations and I already had a CoreOS image loaded up in my account.</p> <p>πŸ‡©πŸ‡ͺ Want CoreOS at Hetzner? There&rsquo;s a <a href="https://major.io/p/deploy-fedora-coreos-in-hetzner-cloud/">blog post</a> for that!</p> <p>Once your CoreOS instance is running, connect to the instance over ssh and ensure the <code>docker.service</code> starts on each boot:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">sudo systemctl <span class="nb">enable</span> --now docker.service </span></span></code></pre></div><p>This ensures that containers come up on each reboot. CoreOS has a podman socket that listens for docker-compatible connections, but that doesn&rsquo;t help with reboots.</p> <p>Perhaps I&rsquo;m old fashioned, but I still enjoy using <a href="https://github.com/docker/compose" target="_blank" rel="noreferrer">docker-compose</a> for container management. I like how I can declare what I want and let <code>docker-compose</code> sort out the rest.</p> <p>Let&rsquo;s install <code>docker-compose</code> on the CoreOS instance now:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="c1"># Check the latest version in the GitHub repo before starting!</span> </span></span><span class="line"><span class="cl"><span class="c1"># https://github.com/docker/compose</span> </span></span><span class="line"><span class="cl">curl -LO https://github.com/docker/compose/releases/download/v2.19.0/docker-compose-linux-x86_64 </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Install docker-compose and make it executable.</span> </span></span><span class="line"><span class="cl">sudo mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose </span></span><span class="line"><span class="cl">sudo chown +x /usr/local/bin/docker-compose </span></span></code></pre></div><p>Verify that <code>docker-compose</code> is ready to go:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> docker-compose --version </span></span><span class="line"><span class="cl"><span class="go">Docker Compose version v2.19.0 </span></span></span></code></pre></div><h1 id="preparing-caddy" class="relative group">Preparing Caddy <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#preparing-caddy" aria-label="Anchor">#</a></span></h1><p>Caddy uses a configuration file called a <em>Caddyfile</em> and we need that in place before we deploy the other containers. Within my home directory, I created a directory called <em>caddy</em>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">mkdir caddy </span></span></code></pre></div><p>Then I added the <em>Caddyfile</em> inside the directory:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-caddy" data-lang="caddy"><span class="line"><span class="cl"><span class="p">{</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> # Your email for LetsEncrypt warnings/notices. </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">email</span> <span class="s">youremail@domain.com</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> # Staging LetsEncrypt server to use while testing. </span></span></span><span class="line"><span class="cl"><span class="c1"> # Uncomment this before going to production! </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">acme_ca</span> <span class="s">https://acme-staging-v02.api.letsencrypt.org/directory</span> </span></span><span class="line"><span class="cl"><span class="p">}</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"># Basic virtual host definition to feed traffic into the </span></span></span><span class="line"><span class="cl"><span class="c1"># Ghost container when it arrives. </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="gh">example.com</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">reverse_proxy</span> <span class="n">ghost</span><span class="p">:</span><span class="mi">2368</span> </span></span><span class="line"><span class="cl"><span class="p">}</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"># OPTIONAL: Redirect traffic to &#39;www&#39; to the bare domain. </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="gh">www.example.com</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="k">redir</span> <span class="s">https://example.com</span><span class="se">{uri}</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span></code></pre></div><p>This configuration sets up LetsEncrypt certificates automatically from the staging server for now. Once we know our configuration is working well, we can comment out the <code>acme_ca</code> line above and get production TLS certificates.</p> <p>At this point, you need a DNS record pointed to your server so you can get a certificate. You have some options:</p> <ul> <li> <p><strong>If the site is entirely new,</strong> just point the root domain name to your CoreOS instance. Use that domain in the configuration above and later in the deployment.</p> </li> <li> <p><strong>If you&rsquo;re migrating from an existing site,</strong> choose a subdomain off your main domain to use. If your website is <em>example.com</em>, use something like <em>test.example.com</em> or <em>new.example.com</em> to get Ghost up and running. It&rsquo;s really easy to change this later.</p> </li> </ul> <p>Now we&rsquo;re ready for the rest of the deployment.</p> <h1 id="deploying-containers" class="relative group">Deploying containers <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#deploying-containers" aria-label="Anchor">#</a></span></h1><p>Here&rsquo;s the <code>docker-compose.yml</code> file I&rsquo;m using:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nn">---</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;3.8&#39;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">services</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># OPTIONAL</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Watchtower monitors all running containers and updates</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># them when the upstream container repo is updated.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">watchtower</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/containrrr/watchtower:latest</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">watchtower</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">unless-stopped</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">hostname</span><span class="p">:</span><span class="w"> </span><span class="l">coreos-ghost-deployment</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">WATCHTOWER_CLEANUP=true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">WATCHTOWER_POLL_INTERVAL=3600</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- --<span class="l">cleanup</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">/var/run/docker.sock:/var/run/docker.sock</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">privileged</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Caddy acts as our external-facing webserver and handles</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># getting TLS certs from LetsEncrypt.</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">caddy</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">caddy:latest</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">caddy</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ghost</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="m">80</span><span class="p">:</span><span class="m">80</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="m">443</span><span class="p">:</span><span class="m">443</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">unless-stopped</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">./caddy/Caddyfile:/etc/caddy/Caddyfile:Z</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ghost:/var/www/html</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">caddy_data:/data</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">caddy_config:/config</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># The Ghost blog software itself</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ghost</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/library/ghost:5</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">ghost</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">always</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">url</span><span class="p">:</span><span class="w"> </span><span class="l">https://example.com</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__client</span><span class="p">:</span><span class="w"> </span><span class="l">mysql</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__host</span><span class="p">:</span><span class="w"> </span><span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__user</span><span class="p">:</span><span class="w"> </span><span class="l">ghost</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__password</span><span class="p">:</span><span class="w"> </span><span class="l">GHOST_PASSWORD_FOR_MARIADB</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">database__connection__database</span><span class="p">:</span><span class="w"> </span><span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ghost:/var/lib/ghost/content</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Our MariaDB database</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ghostdb</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l">docker.io/library/mariadb:11</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">container_name</span><span class="p">:</span><span class="w"> </span><span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">restart</span><span class="p">:</span><span class="w"> </span><span class="l">always</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">MYSQL_ROOT_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l">A_SECURE_ROOT_PASSWORD</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">MYSQL_USER</span><span class="p">:</span><span class="w"> </span><span class="l">ghost</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">MYSQL_PASSWORD</span><span class="p">:</span><span class="w"> </span><span class="l">GHOST_PASSWORD_FOR_MARIADB</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">MYSQL_DATABASE</span><span class="p">:</span><span class="w"> </span><span class="l">ghostdb</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ghostdb:/var/lib/mysql</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">volumes</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">caddy_config</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">caddy_data</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ghost</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ghostdb</span><span class="p">:</span><span class="w"> </span></span></span></code></pre></div><p>I love <a href="https://major.io/p/watchtower/">watchtower</a> but that step is completely optional. It does require some elevated privileges to talk to the podman socket, so <strong>keep that in mind if you choose to use it.</strong></p> <p>Our <code>ghostdb</code> container starts first, followed by <code>ghost</code>, and then <code>caddy</code>. That follows the <code>depends_on</code> configuration keys shown above.</p> <p>There are two steps to take now:</p> <ul> <li>Replace <code>GHOST_PASSWORD_FOR_MARIADB</code> and <code>A_SECURE_ROOT_PASSWORD</code> above with better passwords. πŸ˜‰</li> <li>Also, set the <code>url</code> parameter for the <code>ghost</code> container to your blog&rsquo;s domain name.</li> </ul> <p>Once all of that is done, let&rsquo;s let <code>docker-compose</code> do the heavy lifting:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo docker-compose up -d </span></span></span></code></pre></div><p>Let&rsquo;s verify that our containers are running:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> sudo docker-compose ps </span></span><span class="line"><span class="cl"><span class="go">NAME IMAGE COMMAND SERVICE </span></span></span><span class="line"><span class="cl"><span class="go">caddy caddy:latest &#34;caddy run --config …&#34; caddy </span></span></span><span class="line"><span class="cl"><span class="go">ghost docker.io/library/ghost:5 &#34;docker-entrypoint.s…&#34; ghost </span></span></span><span class="line"><span class="cl"><span class="go">ghostdb docker.io/library/mariadb:11 &#34;docker-entrypoint.s…&#34; ghostdb </span></span></span><span class="line"><span class="cl"><span class="go">watchtower docker.io/containrrr/watchtower:latest &#34;/watchtower --clean…&#34; watchtower </span></span></span></code></pre></div><p>Awesome! πŸ‘</p> <h1 id="ghost-initial-setup" class="relative group">Ghost initial setup <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#ghost-initial-setup" aria-label="Anchor">#</a></span></h1><p>With all of your containers running, browse to <code>https://example.com/ghost/</code> Just add <code>/ghost/</code> to the end of your domain name to reach the admin panel. Create your admin account there with a good password.</p> <p>If everything looks good, run back to your <em>Caddyfile</em> and comment out the <code>acme_ca</code> line:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-caddy" data-lang="caddy"><span class="line"><span class="cl"><span class="p">{</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> # Your email for LetsEncrypt warnings/notices. </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">email</span> <span class="s">youremail@domain.com</span><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> </span></span></span><span class="line"><span class="cl"><span class="c1"> # Staging LetsEncrypt server to use while testing. </span></span></span><span class="line"><span class="cl"><span class="c1"> # Uncomment this before going to production! </span></span></span><span class="line"><span class="cl"><span class="c1"> # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> </span></span></code></pre></div><p>Restart the caddy container to get a production LetsEncrypt certificate on the site:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="go">sudo docker-compose restart caddy </span></span></span></code></pre></div><h1 id="customizing-ghost" class="relative group">Customizing Ghost <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#customizing-ghost" aria-label="Anchor">#</a></span></h1><p>Ghost <a href="https://ghost.org/docs/config/#custom-configuration-files" target="_blank" rel="noreferrer">looks for lots of environment variables</a> to determine its configuration and you can set these in your <code>docker-compose.yml</code> file. Although some configuration items are easy, like <code>url</code>, some are nested and get more complicated. For these, you can use double underscores <code>__</code> to handle the nesting.</p> <p>As an example, we already used <code>database__connection__host</code> in the <code>docker-compose.yaml</code>, and that&rsquo;s the equivalent to this nested configuration:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="s2">&#34;database&#34;</span><span class="err">:</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;connection&#34;</span><span class="p">:</span> <span class="p">{</span> </span></span><span class="line"><span class="cl"> <span class="nt">&#34;host&#34;</span><span class="p">:</span> <span class="s2">&#34;...&#34;</span> </span></span><span class="line"><span class="cl"> <span class="p">}</span> </span></span><span class="line"><span class="cl"><span class="p">}</span> </span></span></code></pre></div><p>If you&rsquo;re deploying in containers, it&rsquo;s a good idea to configure Ghost via environment variables. This ensures that your <code>docker-compose.yml</code> is authoritative for the Ghost deployment. You <em>can</em> <code>exec</code> into the container, adjust the config file on disk, and restart Ghost, but then you have to remember where you configured each item. πŸ₯΅</p> <h1 id="switching-to-production-domain" class="relative group">Switching to production domain <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#switching-to-production-domain" aria-label="Anchor">#</a></span></h1><p>If you used a temporary domain to get everything configured and you&rsquo;re ready to use your production domain, follow these steps:</p> <ul> <li>Open your <em>Caddyfile</em> and replace all instances of the testing domain with the production domain</li> <li>Restart caddy: <code>sudo docker-compose restart caddy</code></li> <li>Edit the <code>docker-compose.yml</code> and change the <code>url</code> key in the <code>ghost</code> container to the production domain</li> <li>Apply the configuration with <code>sudo docker-compose up -d</code></li> </ul> <p>Enjoy your new automatically-updating Ghost blog deployment! πŸ‘»</p>Engineering through layoffshttps://major.io/p/engineering-through-layoffs/Sun, 25 Jun 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/engineering-through-layoffs/<div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300">All comments and thoughts in this post are my own and certainly do not reflect the positions of any of my employers, past and present. The goal of this post is to help with healing after a layoff event and organizing your thoughts around your decisions afterwards.</span> </div> <p>Whatever you want to call them &ndash; layoffs, reductions in force, or downsizing &ndash; they&rsquo;re terrible.</p> <p>For those who leave, uncertainty can become overwhelming. Loss of work means a loss of salary on the simplest level, but it can also mean a loss of purpose. It can mean a loss of critical medical insurance benefits.</p> <p>For those who stay, layoffs shake the foundations of trust with the employer. I have a whole blog post on <a href="https://major.io/p/red-flags">red flags</a> that goes into detail on this topic, so I won&rsquo;t repeat it all here.</p> <p>My argument in this post is that for those who stay and recommit to the mission of the business, <strong>you have much more control over customer outcomes than you ever did before.</strong></p> <h1 id="how-to-think-about-layoffs" class="relative group">How to think about layoffs <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#how-to-think-about-layoffs" aria-label="Anchor">#</a></span></h1><p>I went through several rounds of layoffs at my last employer and my current employer just did a round. Engineers around me often say things like:</p> <ul> <li>&ldquo;How could they let him go? He was so helpful!&rdquo;</li> <li>&ldquo;She is critical to this project and it&rsquo;s our top priority. How could she leave now?&rdquo;</li> <li>&ldquo;He was there for 24 years and he helped everyone, even our CEO. Why would they make him go?&rdquo;</li> <li>&ldquo;Our quarterly results looked great. Why does anyone need to be laid off?&rdquo;</li> </ul> <p>The first step is to avoid <strong>reading deeply into the decision</strong> and avoid <strong>making it personal</strong>.</p> <p>In my experience, some of the decisions are made based on data that you can&rsquo;t see at a publicly traded company. Sometimes a company sheds employees simply because everyone in their market sector is doing it and they&rsquo;re looking for a temporary bump in the stock price. And then there are those situations where a company chooses to end a product line or project.</p> <p>These decisions are often made at a high level within the company and done in such a way to avoid any type of employment-related lawsuits. That brings me to my next topic.</p> <h1 id="why-do-they-let-top-performers-go" class="relative group">Why do they let top performers go? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#why-do-they-let-top-performers-go" aria-label="Anchor">#</a></span></h1><p>This frustrated me many times over the years. As an example, there was a talented network engineer on a team who was a rising star in the company. He could work through complex network topologies to provide a balance of performance and security based on the customer&rsquo;s demands.</p> <p>Even better, he could explain it all to customers. Better still, he could explain it to the customer&rsquo;s technical and non-technical staff.</p> <p>The customer was getting closer to making the deal and this engineer was central to the deal being made. The deal was <strong>large</strong>. Everyone was preparing with implementation calls, documents, and everything else needed for the final meeting.</p> <p><strong>The final meeting came, but the network engineer wasn&rsquo;t on the call.</strong></p> <p>Salespeople, solutions architects, and other engineers were frantic. <em>Did he go home sick?</em> <em>Did we send him the wrong time on the calendar invitation?</em></p> <p>No, as it turns out, he was laid off at lunch and the call was scheduled for 2PM.</p> <p>A choice was made to reduce engineering staff by some percentage, so the business essentially did this:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">FROM</span><span class="w"> </span><span class="n">employees</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">WHERE</span><span class="w"> </span><span class="n">job_family</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&#34;engineering&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">RAND</span><span class="p">()</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">LIMIT</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span></span></span></code></pre></div><p>And they went through the list methodically until their percentage was met.</p> <p><strong>This is why taking layoffs personally will only cause you pain.</strong> Avoid looking for a deeper meaning and explanation where one does not exist.</p> <p>As Yoda once said:</p> <blockquote> <p>Fear is the path to the dark side. Fear leads to anger. Anger leads to hate. Hate leads to suffering.</p> </blockquote> <p>How about the people who aren&rsquo;t top performers?</p> <h1 id="why-dont-companies-just-lay-off-low-performers" class="relative group">Why don&rsquo;t companies just lay off low performers? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#why-dont-companies-just-lay-off-low-performers" aria-label="Anchor">#</a></span></h1><p><em>(Again, try to to avoid looking for deeper meaning here, but I&rsquo;ll go through this question anyway.)</em></p> <p>I&rsquo;ve heard this many times and I&rsquo;ve asked it myself before:</p> <blockquote> <p>Why don&rsquo;t companies just lay off the low performers? After all, some of these people might be toxic to teams and it&rsquo;s clear they&rsquo;re not committed to the company mission.</p> </blockquote> <p>It&rsquo;s a good argument! If a company wants to save money by spending less on salaries and benefits, why not target the people who aren&rsquo;t doing the work first? You&rsquo;d reduce expenses while improving the quality of the workforce!</p> <p>Long story short: <strong>It&rsquo;s not that simple.</strong></p> <p>A wise manager once told me that:</p> <blockquote> <p>If you&rsquo;re the last person to find out that your performance is inadequate, that&rsquo;s not your fault. It&rsquo;s your manager&rsquo;s fault.</p> </blockquote> <p>Managers make mistakes. Whether they&rsquo;re mistakes made on purpose or not, it often opens the company up to litigation.</p> <p>For example, a manager might label an employee as a low performer due to factors outside their job performance. Perhaps they don&rsquo;t look like the rest of the team, they have a different religious affiliation, or a different sexual orientation. They might not participate in after-work functions with the team where alcohol is involved. In some extreme situations, an employee might be labeled a low performer due to rejected romantic advances from the manager. <em>(This last one seems crazy, but I&rsquo;ve seen it happen once.)</em></p> <p>The problem shows up when the company tries to do a layoff like this:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">FROM</span><span class="w"> </span><span class="n">employees</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">WHERE</span><span class="w"> </span><span class="n">performance_level</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&#34;unacceptable&#34;</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">RAND</span><span class="p">()</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">LIMIT</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span></span></span></code></pre></div><p>Suddenly there are people on the low performing list who don&rsquo;t belong there. However, at the executive level, they have no idea about the dubious performance reviews.</p> <p><strong>This is a fast path to wrongful termination lawsuits.</strong></p> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300">First off, be sure that you&rsquo;re ready to recommit to the company mission. If something happened that shook your commitment to the core, take some time to truly understand how you feel about your company. My post on <a href="https://major.io/p/red-flags">red flags</a> might help.</span> </div> <p>Let&rsquo;s get back on something positive. How do we avoid taking these events personally and push through to something better?</p> <h1 id="use-your-newfound-power" class="relative group">Use your newfound power <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#use-your-newfound-power" aria-label="Anchor">#</a></span></h1><p>As an engineer, you have more control over customer outcomes after a layoff than ever before. Confused? I&rsquo;ll explain.</p> <p>I&rsquo;ve worked in engineering, management, and leadership roles in technology since 2004. In many situations, engineers struggle to change business processes and persuade business-minded people to change their outlook on a topic. There&rsquo;s another blog post on here about <a href="https://major.io/p/persuasion-engineering/">persuasion engineering</a> that might be worth reading.</p> <p>Layoffs shake the foundations of any company, including the processes that brought the company to that point. It&rsquo;s a great time to question any of these processes. <em>Does the process save time?</em> <em>Does it benefit customers?</em> <em>Does it need to be modified?</em> <em>Should we throw it away completely?</em></p> <p>I&rsquo;m not suggesting that you approach all processes and business justifications with immediate contempt, but have the courage to ask questions about them. Even long-held beliefs should be questioned.</p> <p>For example, I recently had an exchange like this one:</p> <ul> <li>Me: <em>&ldquo;What if we offered customers the capability to do X?&rdquo;</em></li> <li>Them: <em>&ldquo;Well, we don&rsquo;t have any data to support that.&rdquo;</em></li> <li>Me: <em>&ldquo;This could be an opportunity to guide customers to doing X on our Y product.&rdquo;</em></li> <li>Them: <em>&ldquo;But we need something well defined that customers have asked for before going down that path.&rdquo;</em></li> <li>Me: <em>&ldquo;We&rsquo;ve followed that data for quite some time and the uptake from customers is low. We just went through a round of layoffs &ndash; perhaps we should take a leap here and try something new?&rdquo;</em></li> </ul> <p>The number one fear I have as someone who stays when a layoff happens is this: <em><strong>What if we&rsquo;re too afraid to speak up?</strong></em> <em><strong>What if we&rsquo;re too afraid to take a leap?</strong></em> <em><strong>What if fear of being next on the layoff list prevents us from doing something amazing?</strong></em></p> <p><strong>You can be an advocate for change.</strong> It&rsquo;s the best environment to make a change and think differently about where the company can best serve its customers.</p> <p>It could end in one of two ways:</p> <ol> <li>You change the future of the company for the better and delight your customers</li> <li>You&rsquo;re on the termination list for the next layoff</li> </ol> <p>On the first one, you&rsquo;ve done something truly incredible and you will likely receive recognition for it. You&rsquo;ll also feel more engaged in your work.</p> <p>On the second one, if the company decides you rocked the boat too much and decides to let you go, it&rsquo;s for the best. You&rsquo;re likely dealing with some levels of middle management who lead with fear rather than a drive to improve. Don&rsquo;t take it personally and look for the next opportunity.</p> <p>Personally, I&rsquo;d rather go out in a blaze of glory trying to make the company a better place. πŸ˜‰</p>Launch a watchtower container via podman quadletshttps://major.io/p/podman-quadlet-watchtower/Wed, 31 May 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/podman-quadlet-watchtower/<p>Most of my container workloads run on independent CoreOS cloud instances that I <a href="https://major.io/p/coreos-as-pet/">treat like pets</a>. Keeping containers update remains a constant battle, but it&rsquo;s still easier than running kubernetes.</p> <p>I wrote about <a href="https://major.io/p/watchtower/">using watchtower</a> in the past to keep containers updated. It&rsquo;s a simple container that does a few important things:</p> <ul> <li>It monitors (via docker/podman socket) the running containers on the host</li> <li>It tracks the versions/tags of each container image</li> <li>It looks for updated versions of the container image in their upstream repositories</li> <li>Based on a configurable schedule, it pulls a new container image and restarts the container for updates</li> </ul> <p>I encourage you to <a href="https://github.com/containrrr/watchtower" target="_blank" rel="noreferrer">read more about watchtower on GitHub</a>. There&rsquo;s plenty you can configure, including update intervals, how updates are handled, and how you can get notifications when an update happens.</p> <p>My new deployments always need watchtower running. Luckily, we can combine Fedora CoreOS&rsquo; initial provisioning system, called <a href="https://coreos.github.io/ignition/" target="_blank" rel="noreferrer">ignition</a>, with podman&rsquo;s new <a href="https://www.redhat.com/sysadmin/quadlet-podman" target="_blank" rel="noreferrer">quadlet</a> feature and launch watchtower automatically on the first boot.</p> <h1 id="quadlets" class="relative group">Quadlets <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#quadlets" aria-label="Anchor">#</a></span></h1><p>So what&rsquo;s a quadlet?</p> <p>The <a href="https://www.redhat.com/sysadmin/quadlet-podman" target="_blank" rel="noreferrer">blog post</a> explains it well by making containers more declarative via a familiar systemd syntax. Here&rsquo;s an example <code>.container</code> file from the post:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ini" data-lang="ini"><span class="line"><span class="cl"><span class="k">[Unit]</span> </span></span><span class="line"><span class="cl"><span class="na">Description</span><span class="o">=</span><span class="s">The sleep container</span> </span></span><span class="line"><span class="cl"><span class="na">After</span><span class="o">=</span><span class="s">local-fs.target</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Container]</span> </span></span><span class="line"><span class="cl"><span class="na">Image</span><span class="o">=</span><span class="s">registry.access.redhat.com/ubi9-minimal:latest</span> </span></span><span class="line"><span class="cl"><span class="na">Exec</span><span class="o">=</span><span class="s">sleep 1000</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">[Install]</span> </span></span><span class="line"><span class="cl"><span class="c1"># Start by default on boot</span> </span></span><span class="line"><span class="cl"><span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target default.target</span> </span></span></code></pre></div><p>You can toss this into <code>$HOME/.config/containers/systemd/mysleep.container</code> for rootless user containers or in <code>/etc/containers/systemd/mysleep.container</code> for a container running as root.</p> <h1 id="configure-a-quadlet-on-boot" class="relative group">Configure a quadlet on boot <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#configure-a-quadlet-on-boot" aria-label="Anchor">#</a></span></h1><p>As I mentioned earlier, I want a watchtower container running on my CoreOS nodes at first boot. Let&rsquo;s start with a fairly basic <a href="https://coreos.github.io/butane/" target="_blank" rel="noreferrer">butane</a> file:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">variant</span><span class="p">:</span><span class="w"> </span><span class="l">fcos</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="m">1.4.0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">passwd</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">users</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">major</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">groups</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">wheel</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">sudo</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ssh_authorized_keys</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">storage</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">files</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/containers/systemd/watchtower.container</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">contents</span><span class="p">:</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">inline</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> [Unit] </span></span></span><span class="line"><span class="cl"><span class="sd"> Description=Watchtower container updater </span></span></span><span class="line"><span class="cl"><span class="sd"> Wants=network-online.target </span></span></span><span class="line"><span class="cl"><span class="sd"> After=network-online.target </span></span></span><span class="line"><span class="cl"><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> [Container] </span></span></span><span class="line"><span class="cl"><span class="sd"> ContainerName=watchtower </span></span></span><span class="line"><span class="cl"><span class="sd"> Image=ghcr.io/containrrr/watchtower:1.5.3@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca </span></span></span><span class="line"><span class="cl"><span class="sd"> Environment=WATCHTOWER_CLEANUP=true </span></span></span><span class="line"><span class="cl"><span class="sd"> Environment=WATCHTOWER_POLL_INTERVAL=3600 </span></span></span><span class="line"><span class="cl"><span class="sd"> Volume=/var/run/docker.sock:/var/run/docker.sock </span></span></span><span class="line"><span class="cl"><span class="sd"> SecurityLabelDisable=true </span></span></span><span class="line"><span class="cl"><span class="sd"> </span></span></span><span class="line"><span class="cl"><span class="sd"> [Install] </span></span></span><span class="line"><span class="cl"><span class="sd"> WantedBy=multi-user.target default.target</span><span class="w"> </span></span></span></code></pre></div><p>Let&rsquo;s break this file down:</p> <ol> <li>I start by adding a user named <code>major</code> that has administrative privileges an an ssh key <em>(this is optional, but I like using my own username rather than <code>core</code>)</em></li> <li>The quadlet unit file lands in <code>/etc/containers/systemd/watchtower.container</code> and starts at boot time</li> </ol> <p>The quadlet file has some important configurations:</p> <ol> <li>I added environment variables to clean up outdated container images and check for updates once an hour</li> <li>The podman socket is mounted inside the watchtower container</li> <li>Security labels are disabled to allow for communication with the podman socket</li> </ol> <div class="flex rounded-md bg-primary-100 px-4 py-3 dark:bg-primary-900"> <span class="pe-3 text-primary-400"> <span class="icon relative inline-block px-1 align-text-bottom"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M506.3 417l-213.3-364c-16.33-28-57.54-28-73.98 0l-213.2 364C-10.59 444.9 9.849 480 42.74 480h426.6C502.1 480 522.6 445 506.3 417zM232 168c0-13.25 10.75-24 24-24S280 154.8 280 168v128c0 13.25-10.75 24-23.1 24S232 309.3 232 296V168zM256 416c-17.36 0-31.44-14.08-31.44-31.44c0-17.36 14.07-31.44 31.44-31.44s31.44 14.08 31.44 31.44C287.4 401.9 273.4 416 256 416z"/></svg> </span> </span> <span class="dark:text-neutral-300"><strong>Mounting the podman socket and disabling security labels is not an ideal security approach.</strong> However, I&rsquo;ve found that watchtower&rsquo;s configuration and automation fits my needs really well and I retreive the image from a trusted source. If this won&rsquo;t work for you, you can use <a href="https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html" target="_blank" rel="noreferrer">podman&rsquo;s built-in auto-update</a> feature instead.</span> </div> <p>From here, we convert the butane configuration into an ignition configuration. I&rsquo;m launching this CoreOS node on <a href="https://www.vultr.com/?ref=6941438" target="_blank" rel="noreferrer">VULTR</a>, so I&rsquo;ve named my files accordingly:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> butane vultr-coreos.butane &gt; vultr-coreos.ign </span></span></code></pre></div><h1 id="lets-go-" class="relative group">Let&rsquo;s go πŸš€ <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#lets-go-" aria-label="Anchor">#</a></span></h1><p>I&rsquo;m using VULTR&rsquo;s CLI here in Fedora, but you can do the same steps via VULTR&rsquo;s portal if needed. Just paste in the ignition configuration into the large text box before launch.</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="c1"># Install vultr-cli in Fedora</span> </span></span><span class="line"><span class="cl">sudo dnf install vultr-cli </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="c1"># Launch the instance</span> </span></span><span class="line"><span class="cl">vultr-cli instance create --region dfw --plan vhp-2c-2gb-amd <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --os <span class="m">391</span> --label coreos-dfw-1 --host coreos-dfw-1 <span class="se">\ </span></span></span><span class="line"><span class="cl"><span class="se"></span> --userdata <span class="s2">&#34;</span><span class="k">$(</span>cat vultr-coreos.ign<span class="k">)</span><span class="s2">&#34;</span> </span></span></code></pre></div><p>Let&rsquo;s see how the container is doing:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> ssh major@COREOS_HOST </span></span><span class="line"><span class="cl"><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Fedora CoreOS 38.20230430.3.1 </span></span></span><span class="line"><span class="cl"><span class="go">Tracker: https://github.com/coreos/fedora-coreos-tracker </span></span></span><span class="line"><span class="cl"><span class="go">Discuss: https://discussion.fedoraproject.org/tag/coreos </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="gp">[major@coreos-dfw-1 ~]$</span> sudo podman ps </span></span><span class="line"><span class="cl"><span class="go">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES </span></span></span><span class="line"><span class="cl"><span class="go">a0024712c95d ghcr.io/containrrr/watchtower@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca About a minute ago Up About a minute watchtower </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="gp">[major@coreos-dfw-1 ~]$</span> sudo podman logs watchtower </span></span><span class="line"><span class="cl"><span class="go">time=&#34;2023-05-31T14:01:12Z&#34; level=info msg=&#34;Watchtower 1.5.3&#34; </span></span></span><span class="line"><span class="cl"><span class="go">time=&#34;2023-05-31T14:01:12Z&#34; level=info msg=&#34;Using no notifications&#34; </span></span></span><span class="line"><span class="cl"><span class="go">time=&#34;2023-05-31T14:01:12Z&#34; level=info msg=&#34;Checking all containers (except explicitly disabled with label)&#34; </span></span></span><span class="line"><span class="cl"><span class="go">time=&#34;2023-05-31T14:01:12Z&#34; level=info msg=&#34;Scheduling first run: 2023-05-31 15:01:12 +0000 UTC&#34; </span></span></span><span class="line"><span class="cl"><span class="go">time=&#34;2023-05-31T14:01:12Z&#34; level=info msg=&#34;Note that the first check will be performed in 59 minutes, 59 seconds&#34; </span></span></span></code></pre></div><p><strong>Awesome!</strong> πŸ₯³</p> <p>My system rebooted for an ostree update shortly after provisioning and the container came up automatically both times.</p>CoreOS as a pethttps://major.io/p/coreos-as-pet/Thu, 25 May 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/coreos-as-pet/<p>Anyone working with containers has likely heard of <a href="https://fedoraproject.org/coreos/" target="_blank" rel="noreferrer">CoreOS</a> by this point Haven&rsquo;t heard about it? Don&rsquo;t despair. I&rsquo;ll catch you up on what you missed.</p> <p>Fedora CoreOS offers a really fast pathway to running containers on hardware, in virtual machines, or in clouds. It delivers a lightweight operating system with all of the container technology that you need for running simple containers or launching a kubernetes deployment.</p> <p>But that&rsquo;s not the best part.</p> <p>CoreOS really shines due to its <strong>immutable OS layer</strong><sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>. The OS underneath your containers ships as a single unit and it automatically updates itself much like your mobile phone. An update rolls down, CoreOS sets it up as a secondary OS, reboots into that new update, and rolls back to the original one if there were any issues.</p> <p>Many people use CoreOS as the workhorse underneath kubernetes. Red Hat uses it underneath <a href="https://docs.openshift.com/container-platform/4.13/installing/installing_bare_metal/installing-bare-metal.html#creating-machines-bare-metal_installing-bare-metal" target="_blank" rel="noreferrer">OpenShift</a> as well. It&rsquo;s even supported by the super light weight kubernetes distribution <a href="https://k3s.io/" target="_blank" rel="noreferrer">k3s</a>.</p> <p>But can you use CoreOS as a <em>pet</em> type instance that you use and maintain for long periods of time just like any other server? <strong>Absolutely!</strong></p> <h1 id="whats-this-_pet_-stuff-about" class="relative group">What&rsquo;s this <em>pet</em> stuff about? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#whats-this-_pet_-stuff-about" aria-label="Anchor">#</a></span></h1><p>Whether you like it or not, there&rsquo;s a cattle versus pets paradigm that took hold in the world of IT at some point. The basic ideas are these:</p> <ul> <li>When you take care of cattle, you take care of them as a group. Losing one or more of them would make you sad, but you know you have many others.</li> <li>As for pets, you spend a lot of time taking care of them and playing with them. If you lost one, it would be devastating.</li> </ul> <p>A fleet of web servers could be treated like cattle. Keep lots of them online and replace any instances that have issues.</p> <p>On the other hand, databases or tier zero systems (everyone feels if it they went down) are like pets. You carefully build, maintain, and monitor these.</p> <h1 id="how-does-coreos-fit-in" class="relative group">How does CoreOS fit in? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#how-does-coreos-fit-in" aria-label="Anchor">#</a></span></h1><p>Many people do use CoreOS as a container hosting platform as part of a bigger system. It works really well for that. But it&rsquo;s great as a regular cloud server, too.</p> <p>You can run a single node CoreOS deployment and manage containers via the tools that you know and love. For example, <a href="https://github.com/docker/compose" target="_blank" rel="noreferrer">docker-compose</a> works great on CoreOS. I even used it to host my own <a href="https://major.io/p/self-hosted-mastodon-second-try/">Mastodon deployment</a>.</p> <p>You can also load up more user-friendly tools such as <a href="https://www.portainer.io/" target="_blank" rel="noreferrer">portainer</a> to manage containers in a browser.</p> <h1 id="my-development-tools-are-missing" class="relative group">My development tools are missing! <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-development-tools-are-missing" aria-label="Anchor">#</a></span></h1><p>😱 No <code>vim</code>? This is too minimal! <strong>What are we going to do?</strong></p> <p>Luckily CoreOS comes with <a href="https://github.com/containers/toolbox" target="_blank" rel="noreferrer">toolbox</a>. 🧰</p> <p>Toolbox gives you the ability to run a utility container on the system with some handy benefits:</p> <blockquote> <p>Toolbox environments have seamless access to the user&rsquo;s home directory, the Wayland and X11 sockets, networking (including Avahi), removable devices (like USB sticks), systemd journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc..</p> </blockquote> <p>This means that the toolbox feels like a second OS on the system and it has all of the elevated privileges that you need to do your work. Simply run <code>toolbox enter</code>, follow the prompts, and you&rsquo;ll end up with a Fedora toolbox that matches your CoreOS version. Need a different version, such as Fedora Rawhide? Just specify the Fedora release you want on the prompt:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-console" data-lang="console"><span class="line"><span class="cl"><span class="gp">$</span> toolbox enter --release <span class="m">39</span> </span></span><span class="line"><span class="cl"><span class="go">No toolbox containers found. Create now? [y/N] y </span></span></span><span class="line"><span class="cl"><span class="go">Image required to create toolbox container. </span></span></span><span class="line"><span class="cl"><span class="go">Download registry.fedoraproject.org/fedora-toolbox:39 (500MB)? [y/N]: y </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">Welcome to the Toolbox; a container where you can install and run </span></span></span><span class="line"><span class="cl"><span class="go">all your tools. </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go"> - Use DNF in the usual manner to install command line tools. </span></span></span><span class="line"><span class="cl"><span class="go"> - To create a new tools container, run &#39;toolbox create&#39;. </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">For more information, see the documentation. </span></span></span><span class="line"><span class="cl"><span class="go"></span><span class="err"> </span></span></span><span class="line"><span class="cl"><span class="err"></span><span class="go">β¬’[major@toolbox ~]$ </span></span></span></code></pre></div><p>Look at the <code>toolbox create --help</code> output to see how to create lots of different toolbox containers with different names and releases. If you go overboard and need to delete some toolboxes, just list your toolboxes with <code>toolbox list</code> and follow it up with <code>toolbox rm</code>.</p> <h1 id="my-tool-wont-work-in-the-toolbox" class="relative group">My tool won&rsquo;t work in the toolbox. <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#my-tool-wont-work-in-the-toolbox" aria-label="Anchor">#</a></span></h1><p>Some applications have issues running inside a container, even one that has elevated privileges on the system. CoreOS offers an option for layering packages on top of the underlying immutable OS.</p> <p>Simply run <code>rpm-ostree install PACKAGE</code> to layer a package on top of the OS. When <code>rpm-ostree</code> runs, it creates a new layer and sets that layer to be active on the next boot. That means that you need to reboot before you can use the package.</p> <p>Don&rsquo;t want to reboot? There&rsquo;s another option, but I recommend against it if you can avoid it<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p> <p>You can apply a package layer <em>live</em> on the system without a reboot with the <code>--apply-live</code> flag. Installing a package like <a href="https://github.com/traviscross/mtr" target="_blank" rel="noreferrer">mtr</a> would look like this:</p> <pre tabindex="0"><code>$ sudo rpm-ostree install --apply-live mtr </code></pre><p>As soon as <code>rpm-ostree</code> finishes its work, <code>mtr</code> should be available on the system for you to use.</p> <h1 id="how-do-updates-work" class="relative group">How do updates work? <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#how-do-updates-work" aria-label="Anchor">#</a></span></h1><p>There are two main technologies at work here.</p> <p>First, <a href="https://github.com/coreos/zincati" target="_blank" rel="noreferrer">zincati</a> checks for updates to your immutable OS tree. It runs on a <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/auto-updates/#_os_update_finalization" target="_blank" rel="noreferrer">configurable schedule</a> that you can adjust based on your preferences.</p> <p>Second, <code>rpm-ostree</code> handles the OS layers and switches between them at boot time. If you&rsquo;re running off layer A and an update comes down (layer B), that layer is written to the disk and activated on the next boot. Should there be any issues booting up layer B later, <code>rpm-ostree</code> switches the system back to layer A. In these situations, your downtime might be extended a bit due to two reboots. Your system will come back up with the original OS layer activated.</p> <p>You also get a choice of <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/update-streams/" target="_blank" rel="noreferrer">update streams</a>. Want to live a bit more on the edge? Go for <em>next</em> or <em>testing</em>. You&rsquo;re on the <em>stable</em> stream by default.</p> <p>Although I haven&rsquo;t landed in this situation, it&rsquo;s possible that the system boots into a new update where you notice a problem that doesn&rsquo;t affect the boot. You can <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/auto-updates/#_manual_rollbacks" target="_blank" rel="noreferrer">manually roll back</a> to fix it.</p> <h1 id="i-have-more-questions" class="relative group">I have more questions. <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#i-have-more-questions" aria-label="Anchor">#</a></span></h1><p>Your first stop should be the <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/" target="_blank" rel="noreferrer">Fedora CoreOS docs</a>. There are also lots of ways to <a href="https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/#_getting_in_touch" target="_blank" rel="noreferrer">contact the development team and talk with the community</a>.</p> <p>Love the idea of an immutable OS but you wish you had it for your desktop or laptop? Go check out <a href="https://fedoraproject.org/silverblue/" target="_blank" rel="noreferrer">Fedora Silverblue</a>. πŸ’»</p> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>Okay, so it&rsquo;s <em>mostly</em> immutable. You can edit configuration in <code>/etc</code> and you can layer more packages on top of the base OS layer if you need them. However, CoreOS maintainers discourage adding layered packages if you can avoid it.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:2"> <p>When you apply some packages and make them available immediately, you may lose track of which ones were applied live and which ones are available on the next reboot. Things can get a bit confusing if you suddenly change your mind about applying a package live or not.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>My beef with mailing listshttps://major.io/p/mailing-list-beef/Wed, 10 May 2023 00:00:00 +0000major@mhtx.net (Major Hayden)https://major.io/p/mailing-list-beef/<p>πŸ₯΅ <em><strong>This post is long.</strong> If you need a TL;DR, just <a href="#tldr">hop down to the end</a>.</em></p> <p>One of my toots fell into a <a href="https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/Z2T5WKUTHNLS2SVCVAWNYYRIBK3NGF32/" target="_blank" rel="noreferrer">Fedora Development mailing list discussion</a> recently that was titled <em>&ldquo;It&rsquo;s time to transform the Fedora devel list into something new.&rdquo;</em> As you might imagine, that post blew up.</p> <p>Here&rsquo;s the <a href="https://tootchute.com/@major/109666036733834421" target="_blank" rel="noreferrer">toot</a>:</p> <p> <figure> <picture class="mx-auto my-0 rounded-md" > <img width="580" height="383" class="mx-auto my-0 rounded-md" alt="toot.png" loading="lazy" decoding="async" src="https://major.io/p/mailing-list-beef/toot.png" /> </picture> </figure> </p> <p>After 20 days and 218 emails from 63 participants, the discussion continues. A few people reached out with questions and comments to better understand where I was coming from in that tweet.</p> <p>Long story short: <strong>My beef<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> isn&rsquo;t about the venue, the technology, or the people.</strong></p> <p>My issues are centered around the discourse itself and the time required to parse it.</p> <h1 id="time" class="relative group">Time <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#time" aria-label="Anchor">#</a></span></h1><p>One of my favorite leaders of all time once caught me off guard during a development conversation:</p> <blockquote> <p>Them: Major, what&rsquo;s the most valuable thing you have that you can give to someone else?</p> <p>Me: What I know?</p> <p>Them: No.</p> <p>Me: My ability to understand what they need?</p> <p>Them: No.</p> <p>Me: Okay, just tell me.</p> <p>Them: Your time.</p> </blockquote> <p>He made a point that time is something you can&rsquo;t get back and it&rsquo;s why companies pay people. Companies pay people for their <em>time</em>.</p> <p><em>Time</em> spent doing hard things.</p> <p><em>Time</em> spent away from family.</p> <p><em>Time</em> spent building something new or repairing something that&rsquo;s broken.</p> <p><em>Time</em> spent doing tasks that nobody else wants to do.</p> <p>Humans jump at anything that saves them a little time. We stream Netflix instead of going to the video store or getting DVDs in the mail. We sign up for Amazon Prime to save time on shopping. We rely on appliances to do work for us so we have time for other things.</p> <h1 id="mailing-lists" class="relative group">Mailing lists <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#mailing-lists" aria-label="Anchor">#</a></span></h1><p>Anyone working on open source projects of any scale have likely used mailing lists from time to time. If you&rsquo;re not familiar, here&rsquo;s an example workflow:</p> <ul> <li>You write an email to a special email address with a question, comment, or a request for help</li> <li>That email goes into a system which distributes the email to people who are interested in that topic</li> <li>Mailing list subscribers submit their replies asynchronously until the issue is resolved (or everyone is exhausted)</li> <li>Hopefully you got what you needed</li> </ul> <p><strong>But this post isn&rsquo;t about technology.</strong> You can have asynchronous discussions of varying quality levels just like these in other systems, such as Discourse, forums, Reddit, or bug trackers. There&rsquo;s nothing inherently bad about mailing lists in general. Mailing lists just happen to be very common in open source projects.</p> <p>So what is this post about?</p> <h1 id="unorganized-discourse" class="relative group">Unorganized discourse <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#unorganized-discourse" aria-label="Anchor">#</a></span></h1><p>When someone asks for comments on a topic, especially a controversial one, they get a wide array of replies:</p> <ol> <li>People who don&rsquo;t understand and have questions</li> <li>People who dislike change in all its forms</li> <li>People who dislike your change and provide use cases describing how it&rsquo;s bad</li> <li>People who dislike your change</li> <li>People who heard there&rsquo;s a fight and they can&rsquo;t stay away</li> <li>People who like parts of your idea but want to make changes</li> <li>People who like everything you said<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup></li> <li>People who meant to reply to another thread but replied on yours instead</li> <li>People who are upset about someone else who <a href="https://en.wikipedia.org/wiki/Posting_style#Top-posting" target="_blank" rel="noreferrer">top posted</a> 🀭</li> <li>No reply at all</li> </ol> <p>Filtering through all of these to get to the heart of the argument <strong>is incredibly tedious and time consuming.</strong> πŸ₯΅</p> <p>As Matthew Miller mentioned in his Fedora thread, long threads often cause people to lose the meaning of the discussion. They reply on third, fourth, and fifth level comments in the thread that often veer into other topics. Nobody can rein in the discussion at that point, but people do try. Again, these issues crop up in all kinds of discussion technology systems. They are not unique to mailing lists.</p> <h1 id="improvements" class="relative group">Improvements <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#improvements" aria-label="Anchor">#</a></span></h1><p>The problem is not the mailing list. <strong>It&rsquo;s the discourse.</strong></p> <p>Keeping the discourse more organized is a good option, but its effectiveness is largely governed by the project&rsquo;s communication rules and the civility of those who communicate in the thread. How do we make this better?</p> <h2 id="provide-the-why" class="relative group">Provide the why <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#provide-the-why" aria-label="Anchor">#</a></span></h2><p>I&rsquo;m often amazed at some of the change proposals I see where the technical solution looks so elegant. It&rsquo;s so easy to maintain. It won&rsquo;t be difficult to test. We could implement it so quickly. This person is a genius!</p> <p>Then I stop. Wait a minute. <strong>Why do we need to do this?</strong></p> <p>Always include some of the backstory and use cases behind the change. Ask yourself a few questions:</p> <ul> <li>Why do we need to make the change?</li> <li>Who benefits?</li> <li>Who is harmed?</li> <li>What happens if we don&rsquo;t make the change?</li> <li>Do we have alternatives to the proposed change?</li> <li>Can the change be broken up into smaller pieces?</li> </ul> <p>Include these in your original post to ensure everyone is on a level playing field at the start. This reduces replies requesting more information or questioning the value of the change based on a lack of understanding.</p> <h2 id="intermission-summary" class="relative group">Intermission summary <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#intermission-summary" aria-label="Anchor">#</a></span></h2><p>When I&rsquo;ve written posts that blew up, I took time to read through the replies and summarize the comments thus far. I do this as a reply to my original post<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> with a bulleted list.</p> <p>As an example:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">Thanks for all the replies! So far, this is what I&#39;ve heard </span></span><span class="line"><span class="cl">in the thread: </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">1. Most people think the first change makes sense and should </span></span><span class="line"><span class="cl"> be done soon </span></span><span class="line"><span class="cl">2. The second change requires some more thought based on the </span></span><span class="line"><span class="cl"> use cases provided by lumpynuggets25 </span></span><span class="line"><span class="cl">3. We need better documentation to explain why we are making </span></span><span class="line"><span class="cl"> this change </span></span></code></pre></div><p>This saves time for newcomers to the thread since they can get a brief summary of the comments made thus far without needing to read all of them first. In addition, it gives some of the people who replied a chance to say <em>&ldquo;Yes, that&rsquo;s what I meant. Thank you.&rdquo;</em> It also redirects the conversation back to the original topic and reduces the veering off into other topics.</p> <h2 id="be-patient" class="relative group">Be patient <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#be-patient" aria-label="Anchor">#</a></span></h2><p>I&rsquo;ve seen many threads where the original author obviously felt the need to reply to every comment that came through. This lengthens the thread unnecessarily and causes you to think more about replying rather than understanding the replies. <em>(Other participants in the thread may start doing this themselves.)</em></p> <p>When you send something and the replies start rolling in, just be patient. Sometimes others will engage each other in the thread and answer your questions for you. This gives you time to read and understand what people are saying. You also get the opportunity to build that intermission summary from the previous section. πŸ˜‰</p> <h2 id="let-it-go" class="relative group">Let it go <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#let-it-go" aria-label="Anchor">#</a></span></h2><p>Just like Elsa sang in <a href="https://en.wikipedia.org/wiki/Frozen_%282013_film%29" target="_blank" rel="noreferrer">Frozen</a>, sometimes you have to let it go:</p> <blockquote> <p>Let it go, let it go</p> <p>Can&rsquo;t hold it back anymore</p> <p>Let it go, let it go</p> <p>Turn away and slam the door</p> <p>I don&rsquo;t care what they&rsquo;re going to say</p> <p>Let the storm rage on</p> <p>The cold never bothered me anyway</p> </blockquote> <p>There&rsquo;s always going to be that time where you need to walk away. Take the feedback you received, build on it, and deliver something valuable for people. Let the people who want to be angry for the sake of being angry just be angry on their own.</p> <p>Every thread reaches that point where the people with real comments have provided all of their feedback. Other people just get exhausted with the conversation. That leaves the people who have nothing better to do, and of course, the <a href="https://en.wikipedia.org/wiki/Troll_%28slang%29" target="_blank" rel="noreferrer">trolls</a>. <em>(Don&rsquo;t feed the trolls.)</em></p> <h1 id="tldr" class="relative group">TL;DR <span class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100"><a class="group-hover:text-primary-300 dark:group-hover:text-neutral-700" style="text-decoration-line: none !important;" href="#tldr" aria-label="Anchor">#</a></span></h1><p>Just to summarize:</p> <ul> <li>Mailing lists aren&rsquo;t evil, but unorganized discourse in any medium becomes a time sink and that&rsquo;s evil</li> <li>Always provide the &ldquo;why&rdquo; along with the &ldquo;what&rdquo;</li> <li>Take time to repeat back what was said in a summary</li> <li>Let the comments roll in before considering a limited amount of replies</li> <li>Know when to let it go and build something great</li> </ul> <div class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1"> <p>The word <em>beef</em> is <a href="https://dictionary.cambridge.org/us/dictionary/english/beef" target="_blank" rel="noreferrer">used informally</a> in English as a replacement for <em>complaint</em>.&#160;<a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:2"> <p>Always compliment these people. When people fully agree, they don&rsquo;t reply that often because they feel like they have nothing to add. Sticking your neck out to say &ldquo;This is good and here&rsquo;s why&hellip;&rdquo; is just as treacherous as disagreeing.&#160;<a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:3"> <p>Some people frown on replying to yourself in a mailing list thread. Are you adding value to the thread? If so, ignore them.&#160;<a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </div>