Python on Major Hayden

Coding love for Porkbun

major@mhtx.net (Major Hayden) — Thu, 22 Jan 2026 00:00:00 +0000

Porkbun does exactly what I need and nothing more. I came to them a few years ago with domains scattered across a few different registrars that were quickly becoming too difficult to manage. Their site has minimal upselling, quick renewals, and self-servicing your domains is easy. 🐷

I’m always on the hunt for ways to automate all of my infrastructure, so I set out to build some Porkbun automation. However, many of the python modules for Porkbun only did certain things, such as just DNS or just buying domains. I wanted the whole thing: domain pricing, DNS, DNSSec management. Everything.

Let’s dig into the code. 👨‍💻

Important note: None of these projects are affiliated with Porkbun in any way. They’re just passion projects from someone who really enjoys using their service!

oinker 🐖

My initial project is oinker. It’s a simple Python client (async-first, but sync is there, too) that helps you manage all aspects of your Porkbun account in a type-safe and Pythonic way.

Here’s a quick example:

from oinker import AsyncPiglet, ARecord

async with AsyncPiglet() as piglet:
 # List DNS records
 records = await piglet.dns.list("example.com")
 for record in records:
 print(f"{record.record_type} {record.name} -> {record.content}")

 # Create an A record
 await piglet.dns.create(
 "example.com",
 ARecord(content="1.2.3.4", name="www")
 )

There’s a lot more you can do with it! The documentation covers lots of examples and has an extensive set of API reference material, too.

porkbun-mcp 🤖

LLMs are all around us these days, right? Sometimes I like to talk to Claude via opencode about my DNS records. This gets handy when I’m struggling to find out why one of my kubernetes services isn’t responding and it’s due to a missing DNS record.

There’s porkbun-mcp for that! You can chat with LLMs about your DNS records to get advice, troubleshoot problems, or just create records on the go.

Check the getting started docs to configure porkbun-mcp in your favorite MCP client or LLM tool.

octodns-porkbun 🐙

Finally, there’s octodns-porkbun for those who manage DNS as code with octoDNS. This provider plugin uses oinker under the hood and supports all the record types you’d expect.

If you’re already using octoDNS to manage DNS across multiple providers, this makes it easy to add Porkbun to the mix.

Pull requests welcome!

If you find a way to improve upon any of these projects, please let me know! You can always stop by one of the repos to open an issue, fix a bug, or add a feature!

Fun with docling

major@mhtx.net (Major Hayden) — Fri, 26 Sep 2025 00:00:00 +0000

My team at work does lots of work with retrieval-augmented generation, or RAG, and parsing documents is really painful. It’s so painful that I recently delivered a talk on this very topic at DevConf.US 2025!

Another one of the talks at DevConf.US this year was about docling. We’ve been using it on our team for a while and we really enjoy how it takes challenging documents in various formats and parses them into a single, common document.

I’ll walk you through setting up docling in your project and show you some fun things you can do with it.

Adding docling to a project

I usually use uv for my Python projects. You can start a new project like this:

pipx install uv
mkdir fun-with-docling
cd fun-with-docling
uv init --package --name doclingfun

Now we can add docling to the project. But first, I prefer to install torch-cpu to avoid downloading lots of unnecessary CUDA libraries if I don’t need them.

Start by adding a torch-cpu source in your pyproject.toml file:

[[tool.uv.index]]
name = "torch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

[tool.uv.sources]
torch = { index = "torch-cpu" }
torchvision = { index = "torch-cpu" }

Then add torch, torchvision, and docling to your dependencies:

uv add torch torchvision docling

Verify the installation:

> uv run docling --version
2025-09-26 11:02:48,742 - INFO - Loading plugin 'docling_defaults'
2025-09-26 11:02:48,743 - INFO - Registered ocr engines: ['easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
Docling version: 2.54.0
Docling Core version: 2.48.2
Docling IBM Models version: 3.9.1
Docling Parse version: 4.5.0
Python: cpython-313 (3.13.3)
Platform: Linux-6.16.8-200.fc42.x86_64-x86_64-with-glibc2.41

Parsing a document

Financial markets are a hobby of mine, so I always enjoy reading research on patterns in the market. There’s a great document that we can start with. Download the PDF and parse it with docling:

curl -L -o enhancing_ohlc_data.pdf https://arxiv.org/pdf/2509.16137
> uv run docling enhancing_ohlc_data.pdf --from pdf --to json --output .
2025-09-26 11:07:59,958 - INFO - Loading plugin 'docling_defaults'
2025-09-26 11:07:59,959 - INFO - Registered ocr engines: ['easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
2025-09-26 11:07:59,964 - INFO - paths: [PosixPath('/tmp/tmp243lux62/enhancing_ohlc_data.pdf')]
2025-09-26 11:07:59,964 - INFO - detected formats: [<InputFormat.PDF: 'pdf'>]
2025-09-26 11:07:59,971 - INFO - Going to convert document batch...
2025-09-26 11:07:59,971 - INFO - Initializing pipeline for StandardPdfPipeline with options hash f1301fa0db91f613a1f4baa1a2a11518
2025-09-26 11:07:59,973 - INFO - Loading plugin 'docling_defaults'
2025-09-26 11:07:59,974 - INFO - Registered picture descriptions: ['vlm', 'api']
2025-09-26 11:08:00,241 - INFO - Accelerator device: 'cpu'
2025-09-26 11:08:01,278 - INFO - Accelerator device: 'cpu'
2025-09-26 11:08:05,085 - INFO - Accelerator device: 'cpu'
2025-09-26 11:08:05,381 - INFO - Processing document enhancing_ohlc_data.pdf
2025-09-26 11:08:51,604 - INFO - Finished converting document enhancing_ohlc_data.pdf in 51.64 sec.
2025-09-26 11:08:51,604 - INFO - writing JSON output to enhancing_ohlc_data.json
2025-09-26 11:08:51,667 - INFO - Processed 1 docs, of which 0 failed
2025-09-26 11:08:51,672 - INFO - All documents were converted in 51.71 seconds.

Success! We now have a JSON file that contains the parsed document. (If you’re in a hurry and just want to view the JSON, I’ve uploaded it here.)

Let’s break down how the documents work

Groups and texts

Groups are collections of related content. Here’s an example:

"groups": [
{
 "self_ref": "#/groups/0",
 "parent": {
 "$ref": "#/body"
 },
 "children": [
 {
 "$ref": "#/texts/53"
 },
 {
 "$ref": "#/texts/54"
 },
 {
 "$ref": "#/texts/55"
 },
 {
 "$ref": "#/texts/56"
 }
 ],
 "content_layer": "body",
 "name": "list",
 "label": "list"
},

This group has four children which are called “texts” and we can tell that this is a list of some sort.

If we look for #/texts/53, we can see what the first item in the list is:

{
 "self_ref": "#/texts/53",
 "parent": {
 "$ref": "#/groups/0"
 },
 "children": [],
 "content_layer": "body",
 "label": "list_item",
 "prov": [
 {
 "page_no": 4,
 "bbox": {
 "l": 86.945,
 "t": 669.104,
 "r": 540.004,
 "b": 644.616,
 "coord_origin": "BOTTOMLEFT"
 },
 "charspan": [
 0,
 187
 ]
 }
 ],
 "orig": "\u00b7 Market relevance : VWAP is widely followed by institutional investors, trading algorithms, and benchmark providers [6] [2]. As such, it is a meaningful and valuable quantity to predict.",
 "text": "Market relevance : VWAP is widely followed by institutional investors, trading algorithms, and benchmark providers [6] [2]. As such, it is a meaningful and valuable quantity to predict.",
 "enumerated": false,
 "marker": "\u00b7"
},

This is a single line of text from the document, but it’s part of a list. You can see the parent relationship back to #/groups/0. This text has a label of list_item which indicates that it’s part of a list. There’s also a marker field so you can extract the bullet point character if you want to.

What does this look like in the original document?

So if you found a piece of text that interest you, you can walk backwards to the group and then higher in the document if needed.

Wandering around documents

Let’s break out some Python and see what we can do with documents.

Getting child items

First off, let’s find that "#/groups/0" from earlier and get the texts inside of it:

from docling_core.types.doc.document import DoclingDocument
from rich import print

doc = DoclingDocument.load_from_json("enhancing_ohlc_data.json")

# Extract group 0 and find its children.
my_group = doc.groups[0]
refs = my_group.children

# Loop through the children,
# resolve their location,
# and print the original text.
for x in refs:
 text = x.resolve(doc=doc)
 print(text.orig)

After running this, I get the text from the bulleted list:

· Market relevance : VWAP is widely followed by institutional investors, trading algorithms, and benchmark providers [6] [2]. As such, it is a meaningful and valuable quantity to predict.
· Interval-wide robustness : Unlike point-in-time prices such as the open or close, VWAP summarizes trading activity across the full interval, making it a more stable and representative measure of price.
· Reduced discretization noise : VWAP is a continuous, volume-weighted price and is therefore less affected by the rounding effects of penny-level price changes. In contrast, returns computed from last-trade prices (e.g., close-to-close) are often either exactly zero or
artificially large due to the $0.01 minimum tick increment.
· Weaker arbitrage incentives : Predicting the direction of VWAP changes does not lend itself to straightforward arbitrage. A trader who knows that the next bar's VWAP will be higher than the current one cannot directly profit from this knowledge unless they had already
executed near the current VWAP. As a result, patterns in VWAP returns may persist even in broadly efficient markets.

Getting parent items

What if we wanted to go the other way around? Perhaps there was something really good in #/texts/55 that we wanted to find in the document.

from docling_core.types.doc.document import DoclingDocument
from rich import print

doc = DoclingDocument.load_from_json("enhancing_ohlc_data.json")

# Extract text 55 and get the parent
my_text = doc.texts[55]
parent_ref = my_text.parent

# Resolve the parent reference
parent_item = parent_ref.resolve(doc=doc)
print(parent_item)

Run this and we get the details of #/groups/0:

ListGroup(
 self_ref='#/groups/0',
 parent=RefItem(cref='#/body'),
 children=[RefItem(cref='#/texts/53'), RefItem(cref='#/texts/54'), RefItem(cref='#/texts/55'), RefItem(cref='#/texts/56')],
 content_layer=<ContentLayer.BODY: 'body'>,
 name='list',
 label=<GroupLabel.LIST: 'list'>
)

Removing items

You can also remove individual items from a document or certain classes of items. Let’s assume that you don’t want any lists in your document. You can remove them like this:

from docling_core.types.doc.document import DoclingDocument, ListItem
from rich import print

doc = DoclingDocument.load_from_json("enhancing_ohlc_data.json")

total_texts = len(doc.texts)
print(f"Total texts in document: {total_texts}")

list_items = [x for x in doc.texts if isinstance(x, ListItem)]
doc.delete_items(node_items=list_items)

total_texts_after_deletion = len(doc.texts)
print(f"Total texts after deletion: {total_texts_after_deletion}")

Running this shows that the list items were removed:

> uv run python src/doclingfun/main.py
Total texts in document: 338
Total texts after deletion: 324

Converting to other formats

Finally, you can convert documents to other formats. You can certainly do something simple like this:

doc.save_as_markdown("enhancing_ohlc_data_modified.md")

But what if we wanted our group from earlier serialized to markdown without any other document contents. You can!

from docling_core.transforms.serializer.markdown import MarkdownDocSerializer
from docling_core.types.doc.document import DoclingDocument
from rich import print

doc = DoclingDocument.load_from_json("enhancing_ohlc_data.json")

my_group = doc.groups[0]
serializer = MarkdownDocSerializer(doc=doc)
ser_res = serializer.serialize(item=my_group)
print(ser_res.text)

That results in the same text as before, but now the bullets are removed and replaced with Markdown-style dashes:

- Market relevance : VWAP is widely followed by institutional investors, trading algorithms, and benchmark providers [6] [2]. As such, it is a meaningful and valuable quantity to predict.
- Interval-wide robustness : Unlike point-in-time prices such as the open or close, VWAP summarizes trading activity across the full interval, making it a more stable and representative measure of price.
- Reduced discretization noise : VWAP is a continuous, volume-weighted price and is therefore less affected by the rounding effects of penny-level price changes. In contrast, returns computed from last-trade prices (e.g., close-to-close) are often either exactly zero or
artificially large due to the $0.01 minimum tick increment.
- Weaker arbitrage incentives : Predicting the direction of VWAP changes does not lend itself to straightforward arbitrage. A trader who knows that the next bar's VWAP will be higher than the current one cannot directly profit from this knowledge unless they had already
executed near the current VWAP. As a result, patterns in VWAP returns may persist even in broadly efficient markets.

More to explore

Docling has lots more features that I haven’t covered here. For example, it can do OCR on images. You can also use LLMs to help with parsing. It also comes with a great hybrid chunker to break documents into smaller pieces for embedding into a RAG database.

Mounting the AWS Elastic File Store on Fedora

major@mhtx.net (Major Hayden) — Wed, 13 Sep 2023 00:00:00 +0000

I package a few things here and there in Fedora and one of my latest packages is efs-utils. AWS offers a mount helper for their Elastic File System (EFS) product on GitHub.

In this post, I’ll explain how to:

Launch a Fedora instance on AWS EC2
Install efs-utils and launch the watchdog service
Create an EFS volume in the AWS console
Mount the EFS volume inside the Fedora instance

[!CAUTION] Always check the pricing for any cloud service before you use it! EFS pricing is based on how much you store and how often you access it. Backups are also enabled by default and they add to the monthly charges.

Let’s go! 🚀

Wait, what is EFS?

When you launch a cloud instance (virtual machine) on most clouds, you have different storage options available to you:

Block storage: You can add partitions to this storage, create filesystems, or even use LVM. It looks like someone plugged in a disk to your instance. You get full control over every single storage block on the volume. An example of this is Elastic Block Storage (EBS) on AWS.
Object storage: Although you can’t mount object storage (typically) within your instance, you can read/write objects to this storage via an API. You can upload nearly any type of file you can imagine as an object and then download it later. Objects can also have little bits of metadata attached to them and some of the metadata include prefixes which give a folder-like experience. AWS S3 is a good example of this.
Shared filesystems: This storage shows up in the instance exactly as it sounds: you get a shared filesystem. If you’re familiar with NFS or Samba (SMB), then you’ve used shared filesystems already. They give you much better performance than object storage but offer less freedom than block storage. They’re also great for sharing the same data between multiple instances.

Using EFS is almost like having someone else host a network accessible storage (NAS) device within your cloud deployment.

Launching Fedora

Every image in AWS has an AMI ID attached to it and you need to know the ID for the image you want in your region. You can find these quickly for Fedora by visiting the Fedora Cloud download page. Look for AWS in the list, click the button on that row, and you’ll see a list of Fedora AMI IDs. Click the rocket (🚀) for your preferred region and you’re linked directly to launch that instance in AWS!

I’m clicking the launch link for us-east-2 (Ohio)¹. To finish quickly, I’m choosing all of the default options and using a spot instance (look inside Advanced details at the bottom of the page).

Wait for the instance to finish intializing and access it via ssh:

$ ssh fedora@EXTERNAL_IP
[fedora@ip-172-31-2-38 ~]$ cat /etc/fedora-release 
Fedora release 38 (Thirty Eight)

Success! 🎉

Prepare your security group

Before leaving the EC2 console, you need to make a note of the security group that you used for this instance. That’s because EFS uses security groups to guard access to volumes. Follow these steps to find it:

Click Instances on the left side of the EC2 console.
Click on the row showing the instance we just created.
In the bottom half of the screen, click the Security tab.
Look for Security groups in the security details and copy the security group ID for later.

It should be in the format sg-[a-f0-9]*.

If you click the security group name (after saving it), you’ll see the inbound rules associated with that security group. By default, items in the same security group can’t talk to each other. We need to allow that so our EFS mount will work later.

Click Edit inbound rules and do the following:

Click Add rule.
Choose All traffic in the Type column. (You can narrow this down further later.)
In the source box, look for the security group you just created along with your EC2 instance. If you took the default during the EC2 launch process, it might be named launch-wizard-[0-9]+.
Click Save rules.

Installing efs-utils

Let’s start by getting the efs-utils package onto our new Fedora system:

$ sudo dnf -qy install efs-utils
Installed:
 efs-utils-1.35.0-2.fc38.noarch

The package includes some configuration, a watchdog, and a mount helper:

$ rpm -ql efs-utils
/etc/amazon
/etc/amazon/efs
/etc/amazon/efs/efs-utils.conf
/etc/amazon/efs/efs-utils.crt
/usr/bin/amazon-efs-mount-watchdog
/usr/lib/systemd/system/amazon-efs-mount-watchdog.service
/usr/sbin/mount.efs
/usr/share/doc/efs-utils
/usr/share/doc/efs-utils/CONTRIBUTING.md
/usr/share/doc/efs-utils/README.md
/usr/share/licenses/efs-utils
/usr/share/licenses/efs-utils/LICENSE
/usr/share/man/man8/mount.efs.8.gz
/var/log/amazon/efs

Let’s get the watchdog running so we have that ready later. The watchdog helps to build and tear down the encrypted connection when you mount and unmount an EFS volume:

$ sudo systemctl enable --now amazon-efs-mount-watchdog.service
Created symlink /etc/systemd/system/multi-user.target.wants/amazon-efs-mount-watchdog.service → /usr/lib/systemd/system/amazon-efs-mount-watchdog.service.
$ systemctl status amazon-efs-mount-watchdog.service
● amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog
 Loaded: loaded (/usr/lib/systemd/system/amazon-efs-mount-watchdog.service; enabled; preset: disabled)
 Drop-In: /usr/lib/systemd/system/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Wed 2023-09-13 18:43:46 UTC; 5s ago
 Main PID: 1258 (amazon-efs-moun)
 Tasks: 1 (limit: 4385)
 Memory: 13.3M
 CPU: 76ms
 CGroup: /system.slice/amazon-efs-mount-watchdog.service
 └─1258 /usr/bin/python3 /usr/bin/amazon-efs-mount-watchdog

Sep 13 18:43:46 ip-172-31-2-38.us-east-2.compute.internal systemd[1]: Started amazon-efs-mount-watchdog.service - amazon-efs-mount-watchdog.

Setting up an EFS volume

Start by going over to the EFS console and do the following:

Click File systems in the left navigation bar
Click the orange Create file system button at the top right

A modal appears with a box for the volume name and a VPC selection. Select an easy to remember name (I’m using testing-efs-for-blog-post) and select a VPC. If you’re not sure what a VPC is or which one to use, use the default VPC since that’s likely where your instance landed as well.
Click Create.

There’s a delay while the filesystem initializes and you should see the filesystem show Available with a green check mark after about 30 seconds. Click on the filesystem you just created from the list and you’ll see the details page for the filesystem.

Security setup

EFS volumes come online with the default security group attached and that’s not helpful. From the EFS filesystem details page, click the Network tab and then click Manage.

For each availability zone, go to the Security groups column and add the security group that your instance came up with in the first step. In my case, I accepted the defaults from EC2 and ended up with a launch-wizard-1 security group. Remove the default security group from each. Click Save.

Mounting time

You should still be on the filesystem details page from the previous step. Click Attach at the top right and a modal will appear with mount instructions. The first option should use the EFS mount helper!

For me, it looks like sudo mount -t efs -o tls fs-0baabc62763375bb1:/ efs

Go back to your Fedora instance, create a mount point, and create the volume:

$ sudo mkdir /mnt/efs
$ sudo mount -t efs -o tls fs-0baabc62763375bb1:/ /mnt/efs
$ df -hT | grep efs
127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /mnt/efs

We did it! 🎉

We see 127.0.0.1 here because efs-utils uses stunnel to handle the encryption between your instance and the EFS storage system.

The disk was mounted by root, so we can add a -o user=fedora to give our Fedora user permissions to write files:

$ umount /mnt/efs
$ sudo mount -t efs -o user=fedora,tls fs-0baabc62763375bb1:/ /mnt/efs
$ touch /mnt/efs/test2.txt
$ stat /mnt/efs/test2.txt
 File: /mnt/efs/test2.txt
 Size: 0 	Blocks: 8 IO Block: 1048576 regular empty file
Device: 0,54	Inode: 17657675890899444015 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ fedora) Gid: ( 1000/ fedora)
Context: system_u:object_r:nfs_t:s0
Access: 2023-09-13 19:14:23.308000000 +0000
Modify: 2023-09-13 19:14:23.308000000 +0000
Change: 2023-09-13 19:14:23.308000000 +0000
 Birth: -

Also, efs-utils uses encrypted communication by default, which is great. There may be some situations where you don’t need encrypted communications or you don’t want the overhead. In that case, drop the -o tls option from the mount command and you’ll mount the volume unencrypted.

$ sudo umount /mnt/efs
$ sudo mount -t efs -o user=fedora fs-0baabc62763375bb1:/ /mnt/efs
$ df -hT | grep efs
fs-0baabc62763375bb1.efs.us-east-2.amazonaws.com:/ nfs4 8.0E 0 8.0E 0% /mnt/efs

Extra credit

You can get fancy with access points that allow you to carve up your EFS storage and only let certain instances mount certain parts of the filesystem. So instance A might only be able to mount /files/hr while instance B can only mount /documents.

It would also be a good idea to take an inventory of your security groups and ensure the least amount of instances can reach your EFS volume as possible. Much of the work I did in this post was just for testing. A good plan might be to make a security group for your EFS volume and only allow inbound traffic from security groups which should access it. That would allow you to gather up all of your instances into different security groups and limit access.

Also, be aware of the EFS pricing! 💸

You are billed not only for how much storage you use, but also on requests. Different requests are priced differently depending on access frequency. Backups are also enabled by default at $0.05/GB-month!

Why Ohio? I’m mainly doing it to irritate Corey Quinn. 🤭 Any region you prefer should be fine. ↩︎

Extra icanhazip services going offline

major@mhtx.net (Major Hayden) — Thu, 28 Jul 2022 00:00:00 +0000

Every great thing has its end, and the extra services I launched along with icanhazip.com are no exception. I started icanhazip.com way back in 2009 and detailed much of the history when I transferred ownership to Cloudflare.

The extra services, such as icanhazptr.com, icanhaztrace.com, and icanhaztraceroute.com, came online in 2013 and they weren’t part of the Cloudflare transfer. These services add extra challenges since they need IPv6 connectivity and they don’t play well with containers. Relative to icanhazip.com, these services receive very little traffic.

As much as I’d like to keep running these sites, the extra services will go offline on August 17, 2022.

And if you can’t live without it

Still need PTR record lookups and traceroutes on your network? All of the code is on GitHub in major/icanhaz. To run it, simply execute the icanhaz.py script on your machine.

You can also use gunicorn with a command like this one:

gunicorn icanhaz:app

You can also get very fancy with a systemd unit that exposes a UNIX socket:

[Unit]
Description=Gunicorn instance to serve icanhaz
After=network.target

[Service]
User=nginx
Group=nginx
WorkingDirectory=/opt/icanhaz
ExecStart=/usr/bin/gunicorn --workers 4 --bind unix:icanhaz.sock -m 007 icanhaz:app

[Install]
WantedBy=multi-user.target

And then configure nginx to serve traffic from the socket:

server {
	listen 80;
	listen [::]:80;
	server_name _;
	root /usr/share/nginx/html;


	location / {
		proxy_set_header Host $http_host;
		proxy_pass http://unix:/opt/icanhaz/icanhaz.sock;
	}
}

Thanks for all the support over the last 13 years! 🫂

Install Azure CLI on Fedora 35

major@mhtx.net (Major Hayden) — Mon, 01 Nov 2021 00:00:00 +0000

I started work on packaging the Azure CLI and all of its components in Fedora back in July 2021 and the work finally finished just as the Fedora 35 development cycled ended. This required plenty of packaging work and I was thankful for all the advice I received along the way from experienced Fedora packagers.

Installing Azure CLI

Make sure you’re on Fedora 35 or later first. Then install azure-cli:

$ sudo dnf -y install azure-cli
$ az --version
azure-cli 2.29.0 *

core 2.29.0 *
telemetry 1.0.6

Extensions:
aks-preview 0.5.29

Python location '/usr/bin/python3'
Extensions directory '/home/major/.azure/cliextensions'

Authenticate with Azure

You have two methods for authenticating with Azure:

via a web browser (good for desktops and workstations)
via a device code (good for remote servers or virtual machines)

To authenticate with a browser, type az login and complete the steps in the browser window that appears.

Otherwise, run az login --use-device-code and complete the steps manually using the URL and the access code provided on the command line.

If everything works well, you should get a message saying You have logged in. followed by some information about your account in JSON format.

To the cloud!

Most resources in Azure live inside a resource group, so let’s try to create one to ensure the CLI is working and authenticated properly:

$ az group create --location eastus --resource-group major-testing-eastus
{
 "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/major-testing-eastus",
 "location": "eastus",
 "managedBy": null,
 "name": "major-testing-eastus",
 "properties": {
 "provisioningState": "Succeeded"
 },
 "tags": null,
 "type": "Microsoft.Resources/resourceGroups"
}

Perfect! 🎉

Photo credit: Sergi Marló

A new future for icanhazip

major@mhtx.net (Major Hayden) — Sun, 06 Jun 2021 00:00:00 +0000

In the summer of 2009, I had an idea. My workdays were spent deploying tons of cloud infrastructure as Rackspace acquired Slicehost and we rushed to keep up with the constant demands for new infrastructure from our customers. Working quickly led to challenges with hardware and networking.

That was a time where the I Can Has Cheeseburger meme was red hot just about everywhere. We needed a way to quickly check the public-facing IP address of lots of backend infrastructure and our customers sometimes needed that information, too.

That’s when icanhazip.com was born.

It has always been simple site that returns your external IP address and nothing else. No ads. No trackers. No goofy requirements. Sure, if you looked hard enough, you could spot my attempt at jokes in the HTTP headers. Other than that, the site had a narrow use case and started out mainly as an internal tool.

That’s when things got a little crazy

Lifehacker’s Australian site featured a post about icanhazip.com and traffic went through the roof. My little Slicehost instance was inundated and I quickly realized my Apache and Python setup was not going to work long term.

I migrated to nginx and set up nginx to answer the requests by itself and removed the Python scripts. The load on my small cloud instances came down quickly and I figured the issue would be resolved for a while.

Fast forward to 2015 and icanhazip.com was serving well over 100M requests per day. My cloud instances were getting crushed again, so I deployed more with round robin DNS. (My budget for icanhazip is tiny.) Once that was overloaded, I moved to Hetzner in Germany since I could get physical servers there with better network cards along with unlimited traffic.

The Hetzner servers were not expensive, but I was paying almost $200/month to keep the site afloat and the site made no money. I met some people who worked for Packet.net (now Equinix Metal) and they offered to sponsor the site. This brought my expenses down a lot and I deployed icanhazip.com on one server at Packet.

The site soon crossed 500M requests per day and I deployed a second server. Traffic was still overloading the servers. I didn’t want to spin up more servers at Packet since they were already helping me out quite a bit, so I decided to look under the hood of the kernel and make some improvements.

I learned more than I ever wanted to know about TCP backlogs, TCP/VLAN offloading, packet coalescing, IRQ balancing, and a hundred other things. Some Red Hat network experts helped me (before I joined the company) to continue tweaking. The site was running well after that and I was thankful for the support.

Even crazier still

Soon the site exceeded 1B requests per day. I went back to the people who helped me at Red Hat and after they looked through everything I sent, their response was similar to the well-known line from Jaws: “You’re gonna need a bigger boat.”

I languished on Twitter about how things were getting out of control and someone from Cloudflare reached out to help. We configured Cloudflare to filter traffic in front of the site and this reduced the impact from SYN floods, half-open TLS connections, and other malicious clients that I couldn’t even see when I hosted the site on my own.

Later, Cloudflare launched workers and my contact there said I should consider it since my responses were fairly simple and the workers product would handle it well. The cost for workers looked horrifying at my traffic levels, but the folks at Cloudflare offered to run my workers for free. Their new product was getting bucket loads of traffic and I was able to scale the site even further.

In 2021, the traffic I once received in a month started arriving in 24 hours. The site went from 1B requests per day to 30-35B requests per day over a weekend. Almost all of that traffic came from several network blocks in China. Through all of this, Cloudflare’s workers kept chugging along and my response times barely moved. I was grateful for the help.

Cloudflare was doing a lot for me and I wanted to curb some of the malicious traffic to reduce the load on their products. I tried many times to reach out to the email addresses on the Chinese ASNs and couldn’t make contact with anyone. Some former coworkers told me that my chances of changing that traffic or getting a response to an abuse request was near zero.

Malware almost ended everything

There was a phase for a few years where malware authors kept writing malware that would call out to icanhazip.com to find out what they had infected. If they could find out the external IP address of the systems they had compromised, they could quickly assess the value of the target. Upatre was the first, but many followed after that.

I received emails from companies, US state governments, and even US three letter agencies (TLA). Most were very friendly and they had lots of questions. I explained how the site worked and rarely heard a lot more communication after that.

Not all of the interactions were positive, however. One CISO of a US state emailed me and threatened all kinds of legal action claiming that icanhazip.com was involved in a malware infection in his state’s computer systems. I tried repeatedly to explain how the site worked and that the malware authors were calling out to my site and I was powerless to stop it.

Along the way, many of my hosting providers received abuse emails about the site. I was using a colocation provider in Dallas for a while and the tech called me about an abuse email:

“So we got another abuse email for you,” they said.

“For icanhazip.com?”

“Yes. I didn’t know that was running here, I use it all the time!”

“Thanks! What do we do?”

“Your site just returns IP addresses, right?”

“Yes, that’s it.”

“You know what, I’ll write up a generic response and just start replying to these idiots for you from now on.”

There were many times where I saw a big traffic jump and I realized the traffic was coming from the same ASN, and likely from the same company. I tried reaching out to these companies when I saw it but they rarely ever replied. Some even became extremely hostile to my emails.

The passion left in my passion project started shrinking by the day.

The fun totally dried up

Seeing that over 90% of my traffic load was malicious and abusive was frustrating. Dealing with the abuse emails and complaints was worse.

I built the site originally as just a utility for my team to use, but then it grew and it was fun to find new ways to handle the load without increasing cost. Seeing 2 petabytes of data flowing out per month and knowing that almost all of it was garbage pushed me over the line. I knew I needed a change.

I received a few small offers from various small companies ($5,000 or less), but I realized that the money wasn’t what I was after. I wanted someone to run the site and help the information security industry to stop some of these malicious actors.

icanhazip.com lives on at Cloudflare

I’ve worked closely with my contacts at Cloudflare for a long time and they’ve always jumped in to help me when something wasn’t working well. Their sponsorship of icanhazip.com has saved me tens of thousands of dollars per month. It has also managed to keep the site alive even under horrific traffic load.

I made this decision because Cloudflare has always done right by me and they’ve pledged not only to keep the site running, but to work through the traffic load and determine how to stop the malicious traffic. Their coordinated work with other companies to stop compromised machines from degrading the performance of so many sites was a great selling point for me.

If you’re curious, Cloudflare did pay me for the site. We made a deal for them to pay me $8.03; the cost of the domain registration. The goal was never to make money from the site (although I did get about $75 in total donations from 2009 to 2021). The goal was to provide a service to the internet. Cloudflare has helped me do that and they will continue to do it as the new owners and operators of icanhazip.com.

Gratitude

I’d like to thank everyone who has helped me with icanhazip.com along the way. Tons of people stepped up to help with hosting and server optimization. Hosting providers helped me field an onslaught of abuse requests and DDoS attacks. Most of all, thanks to the people who used the site and helped to promote it.

Photo credit: Sebastien Gabriel on Unsplash

Efficient emojis with rofimoji

major@mhtx.net (Major Hayden) — Sat, 15 May 2021 00:00:00 +0000

Emojis brighten up any message or document. They also serve as excellent methods for testing whether your application handles strings appropriately. (This can be a lot of fun.) 🤭

I constantly obsess with efficiency and shortening the time and effort required to get my work done. I noticed that I could type short text emoticons like :) and ;) so much faster than I could use emojis. This simply would not do. 😉

First attempts

Emoji Copy was my first try at getting the emojis I needed quickly. The site also offers a native emoji mode which allows you to see if your system is handling emojis correctly. The site loads quickly and the search finds emojis in a flash, but it was annoying to open a browser tab just to find an emoji. 🤦🏻‍♂️

The GNOME Extension called Emoji Selector made the selection process faster, but I moved from GNOME to i3 and lost my GNOME extensions. 🤷🏻‍♂️

Other methods, such as the Emoji input method and the ibus-typing-booster, also worked, but I knew there had to be something more efficient than those. 🤔

Enter rofimoji

The rofi launcher quickly become part of my core workflow in i3 (replacing dmenu) and I was pleasantly surprised to find rofimoji in GitHub. 🤗

The rofimoji launcher follows in rofi’s footsteps and gives you quick access to emojis. Using rofimoji is easy:

Bind a key combination to run rofimoji (I use Mod+E)
Type in a search term to find the perfect emoji
Press enter to input it directly in the active window or shift+enter to copy it to the clipboard
🎉

Depending on the application you’re using, you might need to mess around with roflmoji’s --action parameter. Some applications will take the emoji directly as if you typed it from a keyboard, but most of the ones I use seem to like a copy/paste method via the clipboard. 📋

I use the --action clipboard parameter and it works well across browsers and terminals. Here’s the line from the i3 configuration file:

bindsym $mod+e exec --no-startup-id rofimoji --skin-tone light --action clipboard --rofi-args='-theme solarized -font "hack 12" -width 800'

RPMs for Fedora, CentOS, and RHEL

At the moment, rofimoji is not packaged for Fedora, CentOS, or Red Hat Enterprise Linux (RHEL). However, you can install it from my COPR packages repository:

sudo dnf copr enable mhayden/packages
sudo dnf install python3-rofimoji

Enjoy! 🍰

Photo credit: Kelvin Yan on Unsplash

Changes in RHEL 7 Security Technical Implementation Guide Version 1, Release 3

major@mhtx.net (Major Hayden) — Thu, 02 Nov 2017 15:00:25 +0000

The latest release of the Red Hat Enterprise Linux Security Technical Implementation Guide (STIG) was published last week. This release is Version 1, Release 3, and it contains four main changes:

V-77819 - Multifactor authentication is required for graphical logins
V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled
V-77823 - Single user mode must require user authentication
V-77825 - Address space layout randomization (ASLR) must be enabled

Deep dive

Let’s break down this list to understand what each one means.

V-77819 - Multifactor authentication is required for graphical logins

This requirement improves security for graphical logins and extends the existing requirements for multifactor authentication for logins (see V-71965, V-72417, and V-72427). The STIG recommends smartcards (since the US Government often uses CAC cards for multifactor authentication), and this is a good idea for high security systems.

I use Yubikey 4’s as smartcards in most situations and they work anywhere you have available USB slots.

V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled

DCCP is often used as a congestion control mechanism for UDP traffic, but it isn’t used that often in modern networks. There have been vulnerabilities in the past that are mitigated by disabling DCCP, so it’s a good idea to disable it unless you have a strong reason for keeping it enabled.

The ansible-hardening role has been updated to disable the DCCP kernel module by default.

V-77823 - Single user mode must require user authentication

Single user mode is often used in emergency situations where the server cannot boot properly or an issue must be repaired without a fully booted server. This mode can only be used at the server’s physical console, serial port, or via out-of-band management (DRAC, iLO, and IPMI). Allowing single-user mode access without authentication is a serious security risk.

Fortunately, every distribution supported by the ansible-hardening role already has authentication requirements for single user mode in place. The ansible-hardening role does not make any adjustments to the single user mode unit file since any untested adjustment could cause a system to have problems booting.

V-77825 - Address space layout randomization (ASLR) must be enabled

ASLR is a handy technology that makes it more difficult for attackers to guess where a particular program is storing data in memory. It’s not perfect, but it certainly raises the difficulty for an attacker. There are multiple settings for this variable and the kernel documentation for sysctl has some brief explanations for each setting (search for randomize_va_space on the page).

Every distribution supported by the ansible-hardening role is already setting kernel.randomize_va_space=2 by default, which applies randomization for the basic parts of process memory (such as shared libraries and the stack) as well as the heap. The ansible-hardening role will ensure that the default setting is maintained.

ansible-hardening is already up to date

If you’re already using the ansible-hardening role’s master branch, these changes are already in place! Try out the new updates and open a bug report if you find any problems.

Old role, new name: ansible-hardening

major@mhtx.net (Major Hayden) — Tue, 27 Jun 2017 20:49:44 +0000

The interest in the openstack-ansible-security role has taken off faster than I expected, and one piece of constant feedback I received was around the name of the role. Some users were unsure if they needed to use the role in an OpenStack cloud or if the OpenStack-Ansible project was required.

The role works everywhere - OpenStack cloud or not. I started a mailing list thread on the topic and we eventually settled on a new name: ansible-hardening! The updated documentation is already available.

The old openstack-ansible-security role is being retired and it will not receive any additional updates. Moving to the new role is easy:

Install ansible-hardening with ansible-galaxy (or git clone)
Change your playbooks to use the ansible-hardening role

There’s no need to change any variable names or tags - they are all kept the same in the new role.

As always, if you have questions or comments about the role, drop by #openstack-ansible on Freenode IRC or open a bug in Launchpad.

Fixing OpenStack noVNC consoles that ignore keyboard input

major@mhtx.net (Major Hayden) — Thu, 18 May 2017 16:58:56 +0000

I opened up a noVNC console to a virtual machine today in my OpenStack cloud but found that the console wouldn’t take keyboard input. The Send Ctrl-Alt-Del button in the top right of the window worked just fine, but I couldn’t type anywhere in the console. This happened on an Ocata OpenStack cloud deployed with OpenStack-Ansible on CentOS 7.

Test the network path

The network path to the console is a little deep for this deployment, but here’s a quick explanation:

My laptop connects to HAProxy
HAProxy sends the traffic to the nova-novncproxy service
nova-novncproxy connects to the correct VNC port on the right hypervisor

If all of that works, I get a working console! I knew the network path was set up correctly because I could see the console in my browser.

My next troubleshooting step was to dump network traffic with tcpdump on the hypervisor itself. I dumped the traffic on port 5900 (which was the VNC port for this particular instance) and watched the output. Whenever I wiggled the mouse over the noVNC console in my browser, I saw a flurry of network traffic. The same thing happened if I punched lots of keys on the keyboard. At this point, it was clear that the keyboard input was making it to the hypervisor, but it wasn’t being handled correctly.

Test the console

Next, I opened up virt-manager, connected to the hypervisor, and opened a connection to the instance. The keyboard input worked fine there. I opened up remmina and connected via plain old VNC. The keyboard input worked fine there, too!

Investigate in the virtual machine

The system journal in the virtual machine had some interesting output:

kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key pressed (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key pressed (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.

It seems like my keyboard input was being lost in translation - literally. I have a US layout keyboard (Thinkpad X1 Carbon) and the virtual machine was configured with the en-us keymap:

# virsh dumpxml 4 | grep vnc
 <graphics type='vnc' port='5900' autoport='yes' listen='192.168.250.41' keymap='en-us'>

A thorough Googling session revealed that it is not recommended to set a keymap for virtual machines in libvirt in most situations. I set the nova_console_keymap variable in /etc/openstack_deploy/user_variables.yml to an empty string:

nova_console_keymap: ''

I redeployed the nova service using the OpenStack-Ansible playbooks:

openstack-ansible os-nova-install.yml

Once that was done, I powered off the virtual machine and powered it back on. (This is needed to ensure that the libvirt changes go into effect for the virtual machine.)

Great success! The keyboard was working in the noVNC console once again!

Photo credit: Wikipedia

RHEL 7 STIG v1 updates for openstack-ansible-security

major@mhtx.net (Major Hayden) — Wed, 05 Apr 2017 17:46:17 +0000

DISA’s final release of the Red Hat Enterprise Linux (RHEL) 7 Security Technical Implementation Guide (STIG) came out a few weeks ago and it has plenty of improvements and changes. The openstack-ansible-security role has already been updated with these changes.

Quite a few duplicated STIG controls were removed and a few new ones were added. Some of the controls in the pre-release were difficult to implement, especially those that changed parameters for PKI-based authentication.

The biggest challenge overall was the renumbering. The pre-release STIG used an unusual numbering convention: RHEL-07-123456. The final version used the more standardized “V” numbers, such as V-72225. This change required a substantial patch to bring the Ansible role inline with the new STIG release.

All of the role’s documentation is now updated to reflect the new numbering scheme and STIG changes. The key thing to remember is that you’ll need to use --skip-tag with the new STIG numbers if you need to skip certain tasks.

Note: These changes won’t be backported to the stable/ocata branch, so you need to use the master branch to get these changes.

Have feedback? Found a bug? Let us know!

IRC: #openstack-ansible on Freenode IRC
Bugs: LaunchPad
E-mail: openstack-dev@lists.rackspace.com with the subject line [openstack-ansible][security]

Power 8 to the people

major@mhtx.net (Major Hayden) — Thu, 22 Sep 2016 00:00:21 +0000

IBM Edge 2016 is almost over and I’ve learned a lot about Power 8 this week. The performance arguments sound really interesting and some of the choices in AIX’s design seem to make a lot of sense.

However, there’s one remaining barrier for me: Power 8 isn’t really accessible for a tinkerer.

Tinkering?

Google defines tinkering as:

attempt to repair or improve something in a casual or desultory way,

often to no useful effect.

“he spent hours tinkering with the car”

When I come across a new piece of technology, I really enjoy learning how it works. I like to find its strengths and its limitations. I use that information to figure out how I might use the technology later and when I would recommend the technology for someone else to use it.

To me, tinkering is simply messing around with something until I have a better understanding of how it works. Tinkering doesn’t have a finish line. Tinkering may not have a well-defined goal. However, it’s tinkering that leads to a more robust community around a particular technology.

For example, take a look at the Raspberry Pi. There were plenty of other ARM systems on the market before the Pi and there are still a lot of them now. What makes the Pi different is that it’s highly accessible. You can get the newest model for $35 and there are tons of guides for running various operating systems on it. There are even more guides for how to integrate it with other items, such as sprinkler systems, webcams, door locks, and automobiles.

Another example is the Intel NUC. Although the NUC isn’t the most cost-effective way to get an Intel chip on your desk, it’s powerful enough to be a small portable server that you can take with you. This opens up the door for software developers to test code wherever they are (we use them for OpenStack development), run demos at a customer location, or make multi-node clusters that fit in a laptop bag.

What makes Power 8 inaccessible to tinkerers?

One of the first aspects that most people notice is the cost. The S821LC currently starts at around $6,000 on IBM’s site, which is a bit steep for someone who wants to learn a platform.

I’m not saying this server should cost less - the pricing seems quite reasonable when you consider that it comes with dual 8-core Power 8 processors in a 1U form factor. It also has plenty of high speed interconnects ready for GPUs and CAPI chips. With all of that considered, $6,000 for a server like this sounds very reasonable.

There are other considerations as well. A stripped down S821LC with two 8-core CPUs will consume about 406 Watts at 50% utilization. That’s a fair amount of power draw for a tinkerer and I’d definitely think twice about running something like that at home. When you consider the cooling that’s required, it’s even more difficult to justify.

What about AIX?

AIX provides some nice benefits on Power 8 systems, but it’s difficult to access as well. Put “learning AIX” into a Google search and look at the results. The first link is a thread on LinuxQuestions.org where the original poster is given a few options:

Buy some IBM hardware
Get in some legal/EULA gray areas with VMware
Find an old Power 5/6 server that is coming offline at a business that is doing a refresh

Having access to AIX is definitely useful for tinkering, but it could be very useful for software developers. For example, if I write a script in Python and I want to add AIX support, I’ll need access to a system running AIX. It wouldn’t necessarily need to be a system with tons of performance, but it would need the functionality of a basic AIX environment.

Potential solutions

I’d suggest two solutions:

Get AIX into an accessible format, perhaps on a public cloud
Make a more tinker-friendly Power 8 hardware platform

Let’s start with AIX. I’d gladly work with AIX in a public cloud environment where I pay some amount for the virtual machine itself plus additional licensing for AIX. It would still be valuable even if the version of AIX had limiters so that it couldn’t be used for production workloads. I would be able to access the full functionality of a running AIX environment.

The hardware side leads to challenges. However, if it’s possible to do a single Power 8 SMT2 CPU in a smaller form factor, this could become possible. Perhaps these could even be CPUs with some type of defect where one or more cores are disabled. That could reduce cost while still providing the full functionality to someone who wants to tinker with Power 8.

Some might argue that this defeats the point of Power 8 since it’s a high performance, purpose-built chip that crunches through some of the world’s biggest workloads. That’s a totally valid argument.

However, that’s not the point.

The point is to get a fully-functional Power 8 CPU - even if it has serious performance limitations - into the hands of developers who want to do amazing things with it. My hope would be that these small tests will later turn into new ways to utilize POWER systems.

It could also be a way for more system administrators and developers to get experience with AIX. Companies would be able to find more people with a base level of AIX knowledge as well.

Final thoughts

IBM has something truly unique with Power 8. The raw performance of the chip itself is great and the door is open for even more performance through NVlink and CAPI accelerators. These features are game changers for businesses that are struggling to keep up with customer demands. A wider audience could learn about this game-changing technology if it becomes more accessible for tinkering.

Photo credit: Wikipedia

Preventing critical services from deploying on the same OpenStack host

major@mhtx.net (Major Hayden) — Tue, 09 Aug 2016 17:07:35 +0000

OpenStack’s compute service, nova, manages all of the virtual machines within a OpenStack cloud. When you ask nova to build an instance, or a group of instances, nova’s scheduler system determines which hypervisors should run each instance. The scheduler uses filters to figure out where each instance belongs.

However, there are situations where the scheduler might put more than one of your instances on the same host, especially when resources are constrained. This can be a problem when you deploy certain highly available applications, like MariaDB and Galera. If more than one of your database instances landed on the same physical host, a failure of that physical host could take down more than one database instance.

Filters to the rescue

The scheduler offers the ServerGroupAntiAffinityFilter filter for these deployments. This allows a user to create a server group, apply a policy to the group, and then begin adding servers to that group.

If the scheduler filter can’t find a way to fulfill the anti-affinity request (which often happens if the hosts are low on resources), it will fail the entire build transaction with an error. In other words, unless the entire request can be fulfilled, it won’t be deployed.

Let’s see how this works in action on an OpenStack Mitaka cloud deployed with OpenStack-Ansible.

Creating a server group

We can use the openstackclient tool to create our server group:

$ openstack server group create --policy anti-affinity db-production
+----------+--------------------------------------+
| Field | Value |
+----------+--------------------------------------+
| id | cd234914-980a-42f2-b77c-602a7cc0080f |
| members | |
| name | db-production |
| policies | anti-affinity |
+----------+--------------------------------------+

We’ve told nova that we want all of the instances in the db-production group to land on different OpenStack hosts. I’ll copy the id to my clipboard since I’ll need that UUID for the next step.

Adding hosts to the group

My small OpenStack cloud has four hypervisors, so I can add four instances to this server group:

$ openstack server create \
 --flavor m1.small \
 --image "Fedora 24" \
 --nic net-id=bc8895ab-98f7-478f-a54a-36b121f7bb3f \
 --key-name personal_servers \
 --hint "group=cd234914-980a-42f2-b77c-602a7cc0080f" \
 --max 4
 prod-db

This server create command looks fairly standard, but I’ve added the --hint parameter to specify that we want these servers scheduled as part of the group we just created. Also, I’ve requested for four servers to be built at the same time. After a few moments, we should have four servers active:

$ openstack server list --name prod-db -c ID -c Name -c Status
+--------------------------------------+-----------+--------+
| ID | Name | Status |
+--------------------------------------+-----------+--------+
| 7e7a81f3-eb02-4751-93c1-a0de999b8423 | prod-db-4 | ACTIVE |
| b742fb58-8ea4-4e26-bfbf-645a698fbb26 | prod-db-3 | ACTIVE |
| 78c7a43c-4deb-40da-a419-e62db37ab41a | prod-db-2 | ACTIVE |
| 7b8af038-6441-40c0-87c8-0a1acced17a6 | prod-db-1 | ACTIVE |
+--------------------------------------+-----------+--------+

If we check the instances, they should be on different hosts:

$ for i in {1..4}; do openstack server show prod-db-${i} -c hostId -f shell; done
hostid="5fea4e5862f82f051e26caf926fe34bd3a9f1439b08a464f817b4c61"
hostid="65d87faf6d9baa110afa5f2e0308781dde4629142170b2c9af1f090b"
hostid="243f833055303efe838b3233f7ba6e1993fb28895ae11c724f10cc73"
hostid="54df76a1e66bd8585cc3c1f8f38e8f4937456394f2409daf2a8b4c2e"

Success!

If we try to build one more instance, it should fail since the scheduler cannot fulfill the anti-affinity policy applied to server group:

$ openstack server create \
 --flavor m1.small \
 --image "Fedora 24" \
 --nic net-id=bc8895ab-98f7-478f-a54a-36b121f7bb3f \
 --key-name personal_servers \
 --hint "group=cd234914-980a-42f2-b77c-602a7cc0080f" \
 --wait \
 prod-db-5
Error creating server: prod-db-5
Error creating server
$ openstack server show prod-db-5 -c fault -f shell
fault="{u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500...

The scheduler couldn’t find a valid host for a fifth server in the anti-affinity group.

Photo credit: “crowded houses” from jesuscm on Flickr

OpenStack instances come online with multiple network ports attached

major@mhtx.net (Major Hayden) — Wed, 03 Aug 2016 14:40:16 +0000

I ran into an interesting problem recently in my production OpenStack deployment that runs the Mitaka release. On various occasions, instances were coming online with multiple network ports attached, even though I only asked for one network port.

The problem

If I issued a build request for ten instances, I’d usually end up with this:

6 instances with one network port attached
2-3 instances with two network ports attached (not what I want)
1-2 instances with three or four network ports attached (definitely not what I want)

When I examined the instances with multiple network ports attached, I found that one of the network ports would be marked as up while the others would be marked as down. However, the IP addresses associated with those extra ports would still be associated with the instance in horizon and via the nova API. All of the network ports seemed to be fully configured on the neutron side.

Digging into neutron

The neutron API logs are fairly chatty, especially while instances are building, but I found two interesting log lines for one of my instances:

172.29.236.41,172.29.236.21 - - [02/Aug/2016 14:03:11] "GET /v2.0/ports.json?tenant_id=a7b0519330ed481884431102a72dd04c&device_id=05eef1bb-5356-43d9-86c9-4d9854d4d46b HTTP/1.1" 200 2137 0.025282
172.29.236.11,172.29.236.21 - - [02/Aug/2016 14:03:15] "GET /v2.0/ports.json?tenant_id=a7b0519330ed481884431102a72dd04c&device_id=05eef1bb-5356-43d9-86c9-4d9854d4d46b HTTP/1.1" 200 3098 0.027803

There are two requests to create network ports for this instance and neutron is allocating ports to both requests. This would normally be just fine, but I only asked for one network port on this instance.

The IP addresses making the requests are unusual, though. 172.29.236.11 and 172.29.236.41 are two of the hypervisors within my cloud. Why are both of them asking neutron for network ports? Only one of those hypervisors should be building my instance, not both. After checking both hypervisors, I verified that the instance was only provisioned on one of the hosts and not both.

Looking at nova-compute

The instance ended up on the 172.29.236.11 hypervisor once it finished building and the logs on that hypervisor looked fine:

nova.virt.libvirt.driver [-] [instance: 05eef1bb-5356-43d9-86c9-4d9854d4d46b] Instance spawned successfully.

I logged into the 172.29.236.41 hypervisor since it was the one that asked neutron for a port but it never built the instance. The logs there had a much different story:

[instance: 05eef1bb-5356-43d9-86c9-4d9854d4d46b] Instance failed to spawn
Traceback (most recent call last):
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
 yield resources
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
 block_device_info=block_device_info)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2773, in spawn
 admin_pass=admin_password)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3191, in _create_image
 instance, size, fallback_from_host)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6765, in _try_fetch_image_cache
 size=size)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 251, in cache
 *args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 591, in create_image
 prepare_template(target=base, max_size=size, *args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
 return f(*args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in fetch_func_sync
 fetch_func(target=target, *args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 429, in fetch_image
 max_size=max_size)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/images.py", line 120, in fetch_to_raw
 max_size=max_size)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/images.py", line 110, in fetch
 IMAGE_API.download(context, image_href, dest_path=path)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/api.py", line 182, in download
 dst_path=dest_path)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 383, in download
 _reraise_translated_image_exception(image_id)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 682, in _reraise_translated_image_exception
 six.reraise(new_exc, None, exc_trace)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 381, in download
 image_chunks = self._client.call(context, 1, 'data', image_id)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 250, in call
 result = getattr(client.images, method)(*args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/v1/images.py", line 148, in data
 % urlparse.quote(str(image_id)))
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/common/http.py", line 275, in get
 return self._request('GET', url, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/common/http.py", line 267, in _request
 resp, body_iter = self._handle_response(resp)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/common/http.py", line 83, in _handle_response
 raise exc.from_response(resp, resp.content)
ImageNotFound: Image 8feacda9-91fd-48ce-b983-54f7b6de6650 could not be found.

This is one of those occasions where I was glad to find an exception in the log. The image that couldn’t be found is an image I’ve used regularly in the environment before, and I know it exists.

Gandering at glance

First off, I asked glance what it knew about the image:

$ openstack image show 8feacda9-91fd-48ce-b983-54f7b6de6650
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| checksum | 8de08e3fe24ee788e50a6a508235aa64 |
| container_format | bare |
| created_at | 2016-08-03T01:25:34Z |
| disk_format | qcow2 |
| file | /v2/images/8feacda9-91fd-48ce-b983-54f7b6de6650/file |
| id | 8feacda9-91fd-48ce-b983-54f7b6de6650 |
| min_disk | 0 |
| min_ram | 0 |
| name | Fedora 24 |
| owner | a7b0519330ed481884431102a72dd04c |
| properties | description='' |
| protected | False |
| schema | /v2/schemas/image |
| size | 204590080 |
| status | active |
| tags | |
| updated_at | 2016-08-03T01:25:39Z |
| virtual_size | None |
| visibility | public |
+------------------+------------------------------------------------------+

If glance knows about the image, why can’t that hypervisor build an instance with that image? While I was scratching my head, Kevin Carter walked by my desk and joined in the debugging.

He asked about how I had deployed glance and what storage backend I was using. I was using the regular file storage backend since I don’t have swift deployed in the environment. He asked me how many glance nodes I had (I had two) and if I was doing anything to sync the images between the glance nodes.

Then it hit me.

Although both glance nodes knew about the image (since that data is in the database), only one of the glance nodes had the actual image content (the actual qcow2 file) stored. That means that if a hypervisor requests the image from a glance node that knows about the image but doesn’t have it stored, the hypervisor won’t be able to retrieve the image.

Unfortunately, the checks go in this order on the nova-compute side:

Ask glance if this image exists and if this tenant can use it
Configure the network
Retrieve the image

If a hypervisor rolls through steps one and two without issues, but then fails on step 3, the network port will be provisioned but won’t come up on the instance. There’s nothing that cleans up that port in the Mitaka release, so it requires manual intervention.

The fix

As a temporary workaround, I took one of the glance nodes offline so that only one glance node is being used. After hundreds of builds, all of the instances came up with only one network port attached!

There are a few options for long-term fixes.

I could deploy swift and put glance images into swift. That would allow me to use multiple glance nodes with the same swift backend. Another option would be to use an existing swift deployment, such as Rackspace’s Cloud Files product.

Since I’m not eager to deploy swift in my environment for now, I decided to remove the second glance node and reconfigure nova to use only one glance node. That means I’m running with only one glance node and a failure there could be highly annoying. However, that trade-off is fine with me until I can get around to deploying swift.

UPDATE: I’ve opened a bug for nova so that the network ports are cleaned up if the instance fails to build.

Photo credit: Flickr: pascalcharest

Talk recap: The friendship of OpenStack and Ansible

major@mhtx.net (Major Hayden) — Wed, 29 Jun 2016 03:43:21 +0000

The 2016 Red Hat Summit is underway in San Francisco this week and I delivered a talk with Robyn Bergeron earlier today. Our talk, When flexibility met simplicity: The friendship of OpenStack and Ansible, explained how Ansible can reduce the complexity of OpenStack environments without sacrificing the flexibility that private clouds offer.

The talk started at the same time as lunch began and the Partner Pavilion first opened, so we had some stiff competition for attendees’ attention. However, the live demo worked without issues and we had some good questions when the talk was finished.

This post will cover some of the main points from the talk and I’ll share some links for the talk itself and some of the playbooks we ran during the live demo.

IT is complex and difficult

Getting resources for projects at many companies is challenging. OpenStack makes this a little easier by delivering compute, network, and storage resources on demand. However, OpenStack’s flexibility is a double-edged sword. It makes it very easy to obtain virtual machines, but it can be challenging to install and configure.

Ansible reduces some of that complexity without sacrificing flexibility. Ansible comes with plenty of pre-written modules that manage an OpenStack cloud at multiple levels for multiple types of users. Consumers, operators, and deployers can save time and reduce errors by using these modules and providing the parameters that fit their environment.

Ansible and OpenStack

Ansible and OpenStack are both open source projects that are heavily based on Python. Many of the same dependencies needed for Ansible are needed for OpenStack, so there is very little additional software required. Ansible tasks are written in YAML and the user only needs to pass some simple parameters to an existing module to get something done.

Operators are in a unique position since they can use Ansible to perform typical IT tasks, like creating projects and users. They can also assign fine-grained permissions to users with roles via reusable and extensible playbooks. Deployers can use projects like OpenStack-Ansible to deploy a production-ready OpenStack cloud.

Let’s build something

In the talk, we went through a scenario for a live demo. In the scenario, the marketing team needed a new website for a new campaign. The IT department needed to create a project and user for them, and then the marketing team needed to build a server. This required some additional tasks, such as adding ssh keys, creating a security group (with rules) and adding a new private network.

The files from the live demo are up on GitHub:

major/ansible-openstack-summit-demo

In the operator-prep.yml, we created a project and added a user to the project. That user was given the admin role so that the marketing team could have full access to their own project.

From there, we went through the tasks as if we were a member of the marketing team. The marketing.yml playbook went through all of the tasks to prepare for building an instance, actually building the instance, and then adding that instance to the dynamic inventory in Ansible. That playbook also verified the instance was up and performed additional configuration of the virtual machine itself - all in the same playbook.

What’s next?

Robyn shared lots of ways to get involved in the Ansible community. AnsibleFest 2016 is rapidly approaching and the OpenStack Summit in Barcelona is happening this October.

Downloads

The presentation is available in a few formats:

Getting started with gertty

major@mhtx.net (Major Hayden) — Wed, 11 May 2016 13:45:53 +0000

When you’re ready to commit code in an OpenStack project, your patch will eventually land in a Gerrit queue for review. The web interface works well for most users, but it can be challenging to use when you have a large amount of projects to monitor. I recently became a core developer on the OpenStack-Ansible project and I searched for a better solution to handle lots of active reviews.

This is where gertty can help. It’s a console-based application that helps you navigate reviews efficiently. I’ll walk you through the installation and configuration process in the remainder of this post.

Installing gertty

The gertty package is available via pip, GitHub, and various package managers for certain Linux distributions. If you’re on Fedora, just install python-gertty via dnf.

In this example, we will use pip:

pip install gertty

Configuration

You will need a .gertty.yaml file in your home directory for gertty to run. I have an example on GitHub that gives you a good start:

Be sure to change the username and password parts to match your Gerrit username and password. For OpenStack’s gerrit server, you can get these credentials in the user settings area.

Getting synchronized

Now that gertty is configured, start it up on the console:

$ gertty

Type a capital L (SHIFT + L) and wait for the list of projects to appear on the screen. You can choose projects to subscribe to (note that these are different than Gerrit’s watched projects) by pressing your ’s’ key.

However, if you need to follow quite a few projects that match a certain pattern, there’s an easier way. Quit gertty (CTRL - q) and adjust the sqlite database that gertty uses:

$ sqlite3 .gertty.db
SQLite version 3.8.6 2014-08-15 11:46:33
Enter ".help" for usage hints.
sqlite> SELECT count(*) FROM project WHERE name LIKE '%openstack-ansible%';
39
sqlite> UPDATE project SET subscribed=1 WHERE name LIKE '%openstack-ansible%';
sqlite>

In this example, I’ve subscribed to all projects that contain the string openstack-ansible.

I can start gertty once more and wait for it to sync my new projects down to my local workstation. Keep an eye on the Sync: status at the top right of the screen. It will count up as it enumerates reviews to retrieve and then count down as those reviews are downloaded.

You can also create custom dashboards for gertty based on custom queries. In my example configuration file above, I have a special dashboard that contains all OpenStack-Ansible reviews. That dashboard appears whenever I press F5. You can customize these dashboards to include any custom queries that you need for your projects.

Photo credit: Frank Taillandier

Automated Let’s Encrypt DNS challenges with Rackspace Cloud DNS

major@mhtx.net (Major Hayden) — Thu, 31 Mar 2016 19:39:50 +0000

Let’s Encrypt has taken the world by storm by providing free SSL certificates that can be renewed via automated methods. They have issued over 1.4 million certificates since launch in the fall of 2015.

If you are not familiar with how Let’s Encrypt operates, here is an extremely simple explanation:

Create a private key
Make a request for a new certificate
Complete the challenge process
You have a certificate!

That is highly simplified, but there is plenty of detail available on how the whole system works.

One of the most popular challenge methods is HTTP. That involves getting a challenge string from Let’s Encrypt, placing the string at a known URL on your domain, and then waiting for verification of the challenge. The process is quick and Let’s Encrypt provides tools that automate much of the process for you.

A challenger appears

A DNS challenge is available in addition to the HTTP challenge. As you might imagine, this involves creating a DNS record with a string provided by Let’s Encrypt. Once the DNS record is in place, it is verified and certificates are issued. The process goes something like this:

Request a new certificate
Get a challenge string
Add a DNS TXT record on your domain with the challenge string as the data
Wait for DNS records to appear on your DNS server
Let’s Encrypt checks for the DNS record
Clean up the DNS record
Get a certificate

Wrapping automation around this method is often easier than using the HTTP method since it does not require any changes on web servers. If someone has 500 web servers but they change their DNS records through a single API with a DNS provider, it quickly becomes apparent that adding a single DNS record is much easier.

In addition, the HTTP challenge method creates problems for websites which are not entirely publicly accessible yet. A stealth startup or a pre-release site could acquire a certificate without needing to allow any access into the webserver. This is also helpful for sites which will never be public facing, such as those on intranets.

Automating the process

After some research, I stumbled upon a project in GitHub called letsencrypt.sh. The project consists of a bash script that makes all the necessary requests to Let’s Encrypt’s API for requesting and obtaining SSL certificates. However, DNS records are tricky since they are usually managed via an API or other non-trivial methods.

The project provides a hook feature which allows anyone to write a script that receives data and does the necessary DNS adjustments to complete the challenge process. I wrote a hook that interfaces with Rackspace’s Cloud DNS API and handles the creation of DNS records:

GitHub: letsencrypt-rackspace-hook

All of the installation and configuration instructions are in the main README file within the repository. You can begin issuing certificates with DNS challenges in a few minutes.

The hook works like this:

letsencrypt.sh hands off the domain name and a challenge string to the hook
The hook adds a DNS record to Rackspace’s DNS servers via the API
The hook keeps checking to see if the DNS record is publicly accessible
Once the DNS record appears, control is handed back to letsencrypt.sh
letsencrypt.sh tells Let’s Encrypt to verify the challenge
Let’s Encrypt verifies the challenge
The hook cleans up the DNS record and displays the paths to the new certificates and keys.

From there, you can configure your configuration management software to push out the new certificate and keys to your production servers. Let’s Encrypt certificates are currently limited to a 90-day duration, so be sure to configure this automation via a cron job. At the very least, set a calendar reminder for yourself a week or two in advance of the expiration.

Keep in mind that Let’s Encrypt and Rackspace’s DNS service are completely free. Free is a good thing.

Let me know what you think of the script! Feel free to make pull requests or issues if you find bugs. I am still working on some automated testing for the script and I hope to have that available in the next week or two.

Photo Credit: Aphernai via Compfight cc

Segmentation faults with sphinx and pyenv

major@mhtx.net (Major Hayden) — Tue, 09 Feb 2016 14:09:44 +0000

I’m a big fan of the pyenv project because it makes installing multiple python versions a simple process. However, I kept stumbling into a segmentation fault whenever I tried to build documentation with sphinx in Python 2.7.11:

writing output... [100%] unreleased
[app] emitting event: 'doctree-resolved'(<document: <section "current series release notes"...>>, u'unreleased')
[app] emitting event: 'html-page-context'(u'unreleased', 'page.html', {'file_suffix': '.html', 'has_source': True, 'show_sphinx': True, 'last

generating indices... genindex[app] emitting event: 'html-page-context'('genindex', 'genindex.html', {'pathto': <function pathto at 0x7f4279d51230>, 'file_suffix': '.html'
Segmentation fault (core dumped)

I tried a few different versions of sphinx, but the segmentation fault persisted. I did a quick reinstallation of Python 2.7.11 in the hopes that a system update of gcc/glibc was causing the problem:

pyenv install 2.7.11

The same segmentation fault showed up again. After a ton of Google searching, I found that the --enable-shared option allows pyenv to use shared Python libraries at compile time:

env PYTHON_CONFIGURE_OPTS="--enable-shared CC=clang" pyenv install -vk 2.7.11

That worked! I’m now able to run sphinx without segmentation faults.

Enabling kwallet after accidentally disabling it

major@mhtx.net (Major Hayden) — Thu, 28 Jan 2016 16:27:44 +0000

Although I use GNOME 3 as my desktop environment, I prefer KDE’s kwallet service to gnome-keyring for some functions. The user interface is a little easier to use and it’s easier to link up to the keyring module in Python.

Accidentally disabling kwallet

A few errant mouse clicks caused me to accidentally disable the kwalletd service earlier today and I was struggling to get it running again. The daemon is usually started by dbus and I wasn’t entirely sure how to start it properly.

If I start kwalletmanager, I see the kwallet icon in the top bar. However, it’s unresponsive to clicks. Starting kwalletmanager on the command line leads to lots of errors in the console:

kwalletmanager(20406)/kdeui (Wallet): The kwalletd service has been disabled
kwalletmanager(20406)/kdeui (Wallet): The kwalletd service has been disabled
kwalletmanager(20406)/kdeui (Wallet): The kwalletd service has been disabled

Manually running kwalletd in the console wasn’t successful either.

Using kcmshell

KDE provides a utility called kcmshell that allows you to start a configuration panel without running the entire KDE environment. If you disable kwallet accidentally like I did, this will bring up the configuration panel and allow you to re-enable it:

kcmshell4 kwalletconfig

You should see kwallet’s configuration panel appear:

Click on Enable the KDE wallet subsystem and then click OK. Once the window closes, start kwalletmanager and you should be able to access your secrets in kwallet again.

Photo Credit: Wei via Compfight cc

Nobody is using your software project. Now what?

major@mhtx.net (Major Hayden) — Fri, 15 Jan 2016 17:35:48 +0000

Working with open source software is an amazing experience. The collaborative process around creation, refinement, and even maintenance, drives more developers to work on open source software more often. However, every developer finds themselves writing code that very few people actually use.

For some developers, this can be really bothersome. You offer your code up to the world only to find that the world is much less interested than you expected. We see projects that fit the “build it, and they will come” methodology all the time, but it can hurt when our projects don’t have the same impact.

Start by asking yourself a question:

Does it matter?

Many of us write software that has a very limited audience. Perhaps we wrote something that worked around a temporary problem or solved an issue that very few poeple would see. Sometimes we write software to work with a project that doesn’t have a large user base.

In these situations, it often doesn’t matter if other contributors don’t show up to collaborate.

However, if you’re eager to build a community around an open source project, here are some tips that have worked well for me.

Make it approachable

Sites like StackOverflow became immensely popular over time because they provide simple, approachable solutions that normally come with a small amount of explanation. Not all of the code snippets are examples of high quality software development, but that’s not the point here. People can search, review something, and get on their way.

Making software more approachable is completely based on your audience. Complicated software, like the cryptography Python library, has an approach towards experienced software developers who want a robust method for handling cryptographic operations. Compare that to the requests Python library. The developers on that project have an audience of Python developers of all skill levels and they lead off with a simple example and very approachable documentation.

Both of those approaches are very different but extremely effective to their respective audiences.

Once you know your audience, make these changes to make your software more approachable to them:

Describe your project’s sweet spot. what does it do better than every other project?
What does your project not do well? This could clue developers into better projects for their needs or entice them to submit patches for improvements.
How do developers get started? This should include simple ways to install the software, test it after installation, and examples of ways to quickly begin using it.
How do you want to receive improvements? If someone finds a bug or area for improvement, how should they submit it and what should their expectations be?

If you haven’t figured it out already, documentation is required. Projects without documentation are quickly skipped over by most developers for a good reason: if you haven’t taken the time to help people understand how to use your project, why should they take the time to understand it themselves. Projects without documentation are often assumed to be less mature and not production-ready.

Once you make it this far, it’s time to charge your extrovert battery and promote it.

Promoting the project

Some of the best-written software projects with the best documentation often find themselves limited by the fact that nobody knows they exist. This can become a challenging problem to solve because it involves actively reaching out to the audience you’ve identified in the previous steps.

The first step is to do some writing about your software project and what problems it tries to solve. The type of writing and the medium for sharing it is completely up to you.

Some people prefer writing blog posts on their own blog. You may be able to get additional readers by publishing it on external sites, such as Medium, or as a guest author on another site. For example, opensource.com invites guest authors to write about various software projects or solutions provided by open source software. If your project is closely affiliated with another large software project, you may be able to publish a post as a guest author on their project site.

Social media can be helpful if it’s used wisely with the right audience. Your followers must be able to get some value from whatever you link them to in your social media posts. Steer clear of clickbait-type posts and be genuine. If you want to build a community, your integrity is your most important asset.

Technical talks

The most effective method for sharing a project is to do it in person. Yes, this means giving a technical talk to an audience. That means standing in front of people. It’s the kind of thing that make an introvert pause. However, if you care about your project, you can tame that technical talk and make a great connection with your audience.

The return on investment in technical talks often takes the form of a decrescendo. Feedback flows in quickly as soon as the talk is over and gradually decreases over time. The rate of decrease largely depends on the impact you make on your audience. A high-impact, emotionally appealing presentation will yield a long tail of feedback that decreases very slowly. Your project might appear in presentations made by other people and you’ll often get additional feedback and involvement from those talks as well.

Photo Credit: Wanaku via Compfight cc

Automatic package updates with dnf

major@mhtx.net (Major Hayden) — Tue, 12 May 2015 01:22:10 +0000

With Fedora 22’s release date quickly approaching, it’s time to familiarize yourself with dnf. It’s especially important since clean installs of Fedora 22 won’t have yum.

Almost all of the command line arguments are the same but automated updates are a little different. If you’re used to yum-updatesd, then you’ll want to look into dnf-automatic.

Installation

Getting the python code and systemd unit files for automated dnf updates is a quick process:

dnf -y install dnf-automatic

Configuration

There’s only one configuration file to review and most of the defaults are quite sensible. Open up /etc/dnf/automatic.conf with your favorite text editor and review the available options. The only adjustment I made was to change the emit_via option to email as opposed to the stdio.

You may want to change the email_to option if you want to redirect email elsewhere. In my case, I already have an email forward for the root user.

dnf Automation

If you look at the contents of the dnf-automatic package, you’ll find some python code, configuration files, and two important systemd files:

For Fedora 25 and earlier:

# rpm -ql dnf-automatic | grep systemd
/usr/lib/systemd/system/dnf-automatic.service
/usr/lib/systemd/system/dnf-automatic.timer

For Fedora 26 and later:

# rpm -ql dnf-automatic | grep systemd
/usr/lib/systemd/system/dnf-automatic-download.service
/usr/lib/systemd/system/dnf-automatic-download.timer
/usr/lib/systemd/system/dnf-automatic-install.service
/usr/lib/systemd/system/dnf-automatic-install.timer
/usr/lib/systemd/system/dnf-automatic-notifyonly.service
/usr/lib/systemd/system/dnf-automatic-notifyonly.timer

These systemd files are what makes dnf-automatic run. The service file contains the instructions so that systemd knows what to run. The timer file contains the frequency of the update checks (defaults to one day). We need to enable the timer and then start it.

For Fedora 25 and earlier:

systemctl enable dnf-automatic.timer

For Fedora 26 and later:

systemctl enable dnf-automatic-install.timer

Check your work:

# systemctl list-timers *dnf*
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2015-05-12 19:57:30 CDT 23h left Mon 2015-05-11 19:57:29 CDT 14min ago dnf-automatic.timer dnf-automatic.service

The output here shows that the dnf-automatic job last ran at 19:57 on May 11th and it’s set to run at the same time tomorrow, May 12th. Be sure to disable and stop your yum-updatesd service if you still have it running on your system from a previous version of Fedora.

Photo Credit: Outer Rim Emperor via Compfight cc

virt-manager: ‘NoneType’ object has no attribute ‘cpus’

major@mhtx.net (Major Hayden) — Thu, 06 Mar 2014 18:44:58 +0000

After upgrading my Fedora 20 Xen hypervisor to virt-manager 1.0.0, I noticed that I couldn’t open the console or VM details for any of my guests. Running virt-manager --debug gave me the following traceback:

Traceback (most recent call last):
 File "/usr/share/virt-manager/virtManager/engine.py", line 803, in _show_vm_helper
 details = self._get_details_dialog(uri, uuid)
 File "/usr/share/virt-manager/virtManager/engine.py", line 760, in _get_details_dialog
 obj = vmmDetails(con.get_vm(uuid))
 File "/usr/share/virt-manager/virtManager/details.py", line 530, in __init__
 self.init_details()
 File "/usr/share/virt-manager/virtManager/details.py", line 990, in init_details
 for name in [c.model for c in cpu_values.cpus]:
AttributeError: 'NoneType' object has no attribute 'cpus'
[Tue, 04 Mar 2014 22:13:31 virt-manager 21019] DEBUG (error:84) error dialog message:
summary=Error launching details: 'NoneType' object has no attribute 'cpus'
details=Error launching details: 'NoneType' object has no attribute 'cpus'

I opened a bug report and the fix was committed upstream today. If you want to make these updates to your Fedora 20 server before the update package is available, just snag the three RPM’s from koji and install them:

mkdir /tmp/virt-manager
cd /tmp/virt-manager
wget http://kojipkgs.fedoraproject.org/packages/virt-manager/1.0.0/4.fc20/noarch/virt-install-1.0.0-4.fc20.noarch.rpm
wget http://kojipkgs.fedoraproject.org/packages/virt-manager/1.0.0/4.fc20/noarch/virt-manager-1.0.0-4.fc20.noarch.rpm
wget http://kojipkgs.fedoraproject.org/packages/virt-manager/1.0.0/4.fc20/noarch/virt-manager-common-1.0.0-4.fc20.noarch.rpm
yum localinstall *.rpm

UPDATE: Thanks to Cole’s comment below, you can actually pull in the RPM’s using koji directly:

koji download-build virt-manager-1.0.0-4.fc20

Relocating a python virtual environment

major@mhtx.net (Major Hayden) — Sun, 25 Nov 2012 21:27:47 +0000

Python’s virtual environment capability is extremely handy for situations where you don’t want the required modules for a particular python project to get mixed up with your system-wide installed modules. If you work on large python projects (like OpenStack), you’ll find that the applications may require certain versions of python modules to operate properly. If these versions differ from the system-wide python modules you already have installed, you might get unexpected results when you try to run the unit tests.

If you build a virtual environment and inspect the files found within the bin directory of the virtual environment, you’ll find that the first line in the executable scripts is set to use the python version specific to that virtual environment. Here’s an example from a virtual environment containing the OpenStack glance project:

#!/home/major/glance/.venv/bin/python
# EASY-INSTALL-SCRIPT: 'glance==2013.1','glance-api'
__requires__ = 'glance==2013.1'
import pkg_resources
pkg_resources.run_script('glance==2013.1', 'glance-api')

However, what if I wanted to take this virtual environment and place it somewhere else on the server where multiple people could use it? The path in the first line of the scripts in bin will surely break.

The first option is to make the virtual environment relocatable. This can produce unexpected results for some software projects, so be sure to test it out before trying to use it in a production environment.

$ virtualenv --relocatable .venv

A quick check of the same python file now shows this:

#!/usr/bin/env python2.6

import os; activate_this=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'activate_this.py'); execfile(activate_this, dict(__file__=activate_this)); del os, activate_this

# EASY-INSTALL-SCRIPT: 'glance==2013.1','glance-api'

This allows for the path to the activate_this.py script to be determined at runtime and allows you to move your virtual environment wherever you like.

In situations where one script within bin would import another script within bin, things can get a little dicey. These are edge cases, of course, but you can get a similar effect by adjusting the path in the first line of each file within bin to the new location of the virtual environment. If you move the virtual environment again, be sure to alter the paths again with sed.

Keep tabs on OpenStack development with OpenStack Watch on Twitter

major@mhtx.net (Major Hayden) — Fri, 08 Jun 2012 12:19:26 +0000

It’s no secret that I’m a fan of Twitter and OpenStack. I found myself needing a better way to follow the rapid pace of OpenStack development and I figured that a Twitter bot would be a pretty good method for staying up to date.

I’d like to invite you to check out @openstackwatch.

First things first, it’s a completely unofficial project that I worked on during my spare time and it’s not affiliated with OpenStack in any way. If it breaks, it’s most likely my fault.

The bot watches for ticket status changes in OpenStack’s Gerrit server and makes a tweet about the change within a few minutes. Every tweet contains the commit’s project, owner, status, and a brief summary of the change. In addition, you’ll get a link directly to the review page on the Gerrit server. Here’s an example:

Hey! It's Dan!

If you’re not a fan of Twitter, there’s a link to the RSS feed in the bio section, or you can just add this URL to your RSS feed reader:

http://api.twitter.com/1/statuses/user_timeline.rss?screen_name=openstackwatch

If you can come up with any ideas for improvements, please let me know!

mysql-json-bridge: a simple JSON API for MySQL

major@mhtx.net (Major Hayden) — Thu, 29 Mar 2012 02:34:53 +0000

My quest to get better at Python led me to create a new project on GitHub. It’s called mysql-json-bridge and it’s ready for you to use.

Why do we need a JSON API for MySQL?

The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That’s where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?

It all starts with a simple HTTP POST. I’ve become a big fan of the Python requests module. If you’re using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they’re converted to JSON and returned via the same HTTP connection.

What technology does it use?

Flask does the heavy lifting for the HTTP requests and Facebook’s Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?

Yes. I haven’t done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You’re terrible at Python.

All feedback (and every pull request) is welcome. I’m still getting the hang of Python (hey, I’ve only been writing in it seriously for a few weeks!) and I’m always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.