Icanhazip on Major Hayden

Extra icanhazip services going offline

major@mhtx.net (Major Hayden) — Thu, 28 Jul 2022 00:00:00 +0000

Every great thing has its end, and the extra services I launched along with icanhazip.com are no exception. I started icanhazip.com way back in 2009 and detailed much of the history when I transferred ownership to Cloudflare.

The extra services, such as icanhazptr.com, icanhaztrace.com, and icanhaztraceroute.com, came online in 2013 and they weren’t part of the Cloudflare transfer. These services add extra challenges since they need IPv6 connectivity and they don’t play well with containers. Relative to icanhazip.com, these services receive very little traffic.

As much as I’d like to keep running these sites, the extra services will go offline on August 17, 2022.

And if you can’t live without it

Still need PTR record lookups and traceroutes on your network? All of the code is on GitHub in major/icanhaz. To run it, simply execute the icanhaz.py script on your machine.

You can also use gunicorn with a command like this one:

gunicorn icanhaz:app

You can also get very fancy with a systemd unit that exposes a UNIX socket:

[Unit]
Description=Gunicorn instance to serve icanhaz
After=network.target

[Service]
User=nginx
Group=nginx
WorkingDirectory=/opt/icanhaz
ExecStart=/usr/bin/gunicorn --workers 4 --bind unix:icanhaz.sock -m 007 icanhaz:app

[Install]
WantedBy=multi-user.target

And then configure nginx to serve traffic from the socket:

server {
	listen 80;
	listen [::]:80;
	server_name _;
	root /usr/share/nginx/html;


	location / {
		proxy_set_header Host $http_host;
		proxy_pass http://unix:/opt/icanhaz/icanhaz.sock;
	}
}

Thanks for all the support over the last 13 years! 🫂

icanhazip.com FAQ

major@mhtx.net (Major Hayden) — Mon, 05 Jul 2021 00:00:00 +0000

icanhazip.com has a new owner!

Starting in June 2021, icanhazip.com is now owned and operated by Cloudflare! Read more about it in the blog post: A new future for icanhazip.

A new future for icanhazip

major@mhtx.net (Major Hayden) — Sun, 06 Jun 2021 00:00:00 +0000

In the summer of 2009, I had an idea. My workdays were spent deploying tons of cloud infrastructure as Rackspace acquired Slicehost and we rushed to keep up with the constant demands for new infrastructure from our customers. Working quickly led to challenges with hardware and networking.

That was a time where the I Can Has Cheeseburger meme was red hot just about everywhere. We needed a way to quickly check the public-facing IP address of lots of backend infrastructure and our customers sometimes needed that information, too.

That’s when icanhazip.com was born.

It has always been simple site that returns your external IP address and nothing else. No ads. No trackers. No goofy requirements. Sure, if you looked hard enough, you could spot my attempt at jokes in the HTTP headers. Other than that, the site had a narrow use case and started out mainly as an internal tool.

That’s when things got a little crazy

Lifehacker’s Australian site featured a post about icanhazip.com and traffic went through the roof. My little Slicehost instance was inundated and I quickly realized my Apache and Python setup was not going to work long term.

I migrated to nginx and set up nginx to answer the requests by itself and removed the Python scripts. The load on my small cloud instances came down quickly and I figured the issue would be resolved for a while.

Fast forward to 2015 and icanhazip.com was serving well over 100M requests per day. My cloud instances were getting crushed again, so I deployed more with round robin DNS. (My budget for icanhazip is tiny.) Once that was overloaded, I moved to Hetzner in Germany since I could get physical servers there with better network cards along with unlimited traffic.

The Hetzner servers were not expensive, but I was paying almost $200/month to keep the site afloat and the site made no money. I met some people who worked for Packet.net (now Equinix Metal) and they offered to sponsor the site. This brought my expenses down a lot and I deployed icanhazip.com on one server at Packet.

The site soon crossed 500M requests per day and I deployed a second server. Traffic was still overloading the servers. I didn’t want to spin up more servers at Packet since they were already helping me out quite a bit, so I decided to look under the hood of the kernel and make some improvements.

I learned more than I ever wanted to know about TCP backlogs, TCP/VLAN offloading, packet coalescing, IRQ balancing, and a hundred other things. Some Red Hat network experts helped me (before I joined the company) to continue tweaking. The site was running well after that and I was thankful for the support.

Even crazier still

Soon the site exceeded 1B requests per day. I went back to the people who helped me at Red Hat and after they looked through everything I sent, their response was similar to the well-known line from Jaws: “You’re gonna need a bigger boat.”

I languished on Twitter about how things were getting out of control and someone from Cloudflare reached out to help. We configured Cloudflare to filter traffic in front of the site and this reduced the impact from SYN floods, half-open TLS connections, and other malicious clients that I couldn’t even see when I hosted the site on my own.

Later, Cloudflare launched workers and my contact there said I should consider it since my responses were fairly simple and the workers product would handle it well. The cost for workers looked horrifying at my traffic levels, but the folks at Cloudflare offered to run my workers for free. Their new product was getting bucket loads of traffic and I was able to scale the site even further.

In 2021, the traffic I once received in a month started arriving in 24 hours. The site went from 1B requests per day to 30-35B requests per day over a weekend. Almost all of that traffic came from several network blocks in China. Through all of this, Cloudflare’s workers kept chugging along and my response times barely moved. I was grateful for the help.

Cloudflare was doing a lot for me and I wanted to curb some of the malicious traffic to reduce the load on their products. I tried many times to reach out to the email addresses on the Chinese ASNs and couldn’t make contact with anyone. Some former coworkers told me that my chances of changing that traffic or getting a response to an abuse request was near zero.

Malware almost ended everything

There was a phase for a few years where malware authors kept writing malware that would call out to icanhazip.com to find out what they had infected. If they could find out the external IP address of the systems they had compromised, they could quickly assess the value of the target. Upatre was the first, but many followed after that.

I received emails from companies, US state governments, and even US three letter agencies (TLA). Most were very friendly and they had lots of questions. I explained how the site worked and rarely heard a lot more communication after that.

Not all of the interactions were positive, however. One CISO of a US state emailed me and threatened all kinds of legal action claiming that icanhazip.com was involved in a malware infection in his state’s computer systems. I tried repeatedly to explain how the site worked and that the malware authors were calling out to my site and I was powerless to stop it.

Along the way, many of my hosting providers received abuse emails about the site. I was using a colocation provider in Dallas for a while and the tech called me about an abuse email:

“So we got another abuse email for you,” they said.

“For icanhazip.com?”

“Yes. I didn’t know that was running here, I use it all the time!”

“Thanks! What do we do?”

“Your site just returns IP addresses, right?”

“Yes, that’s it.”

“You know what, I’ll write up a generic response and just start replying to these idiots for you from now on.”

There were many times where I saw a big traffic jump and I realized the traffic was coming from the same ASN, and likely from the same company. I tried reaching out to these companies when I saw it but they rarely ever replied. Some even became extremely hostile to my emails.

The passion left in my passion project started shrinking by the day.

The fun totally dried up

Seeing that over 90% of my traffic load was malicious and abusive was frustrating. Dealing with the abuse emails and complaints was worse.

I built the site originally as just a utility for my team to use, but then it grew and it was fun to find new ways to handle the load without increasing cost. Seeing 2 petabytes of data flowing out per month and knowing that almost all of it was garbage pushed me over the line. I knew I needed a change.

I received a few small offers from various small companies ($5,000 or less), but I realized that the money wasn’t what I was after. I wanted someone to run the site and help the information security industry to stop some of these malicious actors.

icanhazip.com lives on at Cloudflare

I’ve worked closely with my contacts at Cloudflare for a long time and they’ve always jumped in to help me when something wasn’t working well. Their sponsorship of icanhazip.com has saved me tens of thousands of dollars per month. It has also managed to keep the site alive even under horrific traffic load.

I made this decision because Cloudflare has always done right by me and they’ve pledged not only to keep the site running, but to work through the traffic load and determine how to stop the malicious traffic. Their coordinated work with other companies to stop compromised machines from degrading the performance of so many sites was a great selling point for me.

If you’re curious, Cloudflare did pay me for the site. We made a deal for them to pay me $8.03; the cost of the domain registration. The goal was never to make money from the site (although I did get about $75 in total donations from 2009 to 2021). The goal was to provide a service to the internet. Cloudflare has helped me do that and they will continue to do it as the new owners and operators of icanhazip.com.

Gratitude

I’d like to thank everyone who has helped me with icanhazip.com along the way. Tons of people stepped up to help with hosting and server optimization. Hosting providers helped me field an onslaught of abuse requests and DDoS attacks. Most of all, thanks to the people who used the site and helped to promote it.

Photo credit: Sebastien Gabriel on Unsplash

You have a problem and icanhazip.com isn’t one of them

major@mhtx.net (Major Hayden) — Wed, 20 May 2015 12:50:41 +0000

I really enjoy operating icanhazip.com and the other domains. It’s fun to run some really busy services and find ways to reduce resource consumption and the overall cost of hosting.

My brain has a knack for optimization and improving the site is quite fun for me. So much so that I’ve decided to host all of icanhazip.com out of my own pocket starting today.

However, something seriously needs to change.

A complaint came in yesterday from someone who noticed that their machines were making quite a few requests to icanhazip.com. It turns out there was a problem with malware and the complaint implicated my site as part of the problem. One of my nodes was taken down as a precaution while I furiously worked to refute the claims within the complaint. Although the site stayed up on other nodes, it was an annoyance for some and I received a few tweets and emails about it.

Long story short, if you’re sending me or my ISP a complaint about icanhazip.com, there’s one thing you need to know: the problem is on your end, not mine. Either you have users making legitimate requests to my site or you have malware actively operating on your network.

No, it’s not time to panic.

You can actually use icanhazip.com as a tool to identify problems on your network.

For example, add rules to your intrusion detection systems (IDS) to detect requests to the site in environments where you don’t expect those requests to take place. Members of your support team might use the site regularly to test things but your Active Directory server shouldn’t start spontaneously talking to my site overnight. That’s a red flag and you can detect it easily.

Also, don’t report the site as malicious or hosting malware when it’s not. I’ve been accused of distributing malware and participating in attacks but then, after further investigation, it was discovered that I was only returning an IPv4 address to a valid request. That hardly warrants the blind accusations that I often receive.

I’ve taken some steps to ensure that there’s a way to contact me with any questions or concerns you might have. For example:

You can email abuse, postmaster, and security at icanhazip.com anytime
There’s a HTTP header with a link to the FAQ (which has been there for years)
I monitor any tweets or blog posts that are written about the site

As always, if you have questions or concerns, please reach out to me and read the FAQ. Thanks to everyone for all the support!

Photo Credit: Amir Kamran via Compfight cc

Docker, trusted builds, and Fedora 20

major@mhtx.net (Major Hayden) — Wed, 26 Mar 2014 05:17:58 +0000

Docker is a hot topic in the Linux world at the moment and I decided to try out the new trusted build process. Long story short, you put your Dockerfile along with any additional content into your GitHub repository, link your GitHub account with Docker, and then fire off a build. The Docker index labels it as “trusted” since it was build from source files in your repository.

I set off to build a Dockerfile to provision a container that would run all of the icanhazip services. Getting httpd running was a little tricky, but I soon had a working Dockerfile that built and ran successfully on Fedora 20.

The trusted build process kicked off without much fuss and I found myself waiting for a couple of hours for my job to start. I was sad to see an error after waiting so long:

Installing : httpd-2.4.7-3.fc20.x86_64
error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file

Well, that’s weird. It turns out that cap_set_file is part of libcap that sets filesystem capabilities based on the POSIX.1e standards. You can read up on capabilities in the Linux kernel capabilities FAQ. (Special thanks to Andrew Clayton getting me pointed in the right direction there.)

Marek Goldmann ran into this problem back in September 2013 and opened a bug report. Marek proposed a change to the Docker codebase that would remove setfcap from the list of banned capabilities in the LXC template used by docker. Another workaround would be to use the -privileged option to perform a build in privileged mode (available in docker 0.6+).

Both of those workarounds are unavailable when doing trusted builds with docker’s index. Sigh.

I fired off an email to Docker’s support staff and received a quick reply:

Major,

We are aware of this issue, and we are currently working on a fix, and we hope to have something we can start testing this week. I’m not sure when we will be able to roll out the fix, but we are hoping soon. Until then, there isn’t anything you can do to work around it. Sorry for the inconvenience.

If anything changes, we will be sure to let you know.

Ken

It wasn’t the answer I wanted but it’s good to know that the issue is being worked. In the meantime, I’ll push an untrusted build of the icanhazip Docker container up to the index for everyone to enjoy.

Stay tuned for updates.

UPDATED 2014-08-08: Per Thomas’ comment below, this has been fixed upstream.

Puppy Linux, icanhazip, and tin foil hats

major@mhtx.net (Major Hayden) — Mon, 10 Feb 2014 04:04:27 +0000

I figured that the Puppy Linux and icanhazip.com fiasco was over, but I was wrong:

@majorhayden you're in Puppy Linux controversy again http://t.co/B21JPIx7Ob #Heat

— Michael Amadio (@01micko) January 14, 2014

After a quick visit to the forums, I found the debate stirred up again. Various users were wondering if their internet connections were somehow compromised or if a remote American network was somehow spying on their internet traffic. Others wondered if some secretive software was added to the Puppy Linux distribution that was calling out to the site.

Fortunately, quite a few users on the forum showed up to explain that Puppy Linux has a built-in feature to figure out a user’s external IP address to help them get started with their system after it boots. Another user was kind enough to dig up the Lifehacker post about icanhazip from 2011.

Many users on the forum were still dissatisfied. Many of them turned their questions to maintainers of the distribution (which is where those questions should go), but many others felt that icanhazip was the source of the problem. Some of them felt so strongly that they called my hosting provider via telephone to curse at them. Here’s a snippet of an email I received from my colocation provider:

I had an interesting call from someone today said that 216.69.252.101 was showing up on his computer. Sounded kind of [omitted] and called me a *\* ******…

Let’s get three things straight:

I’m a huge supporter of everything Linux, including Puppy Linux. I don’t hold a grudge against the project for what a minority of their users do.
I don’t collect data when users visit icanhazip.com other than standard Apache logs. No cookies are used.
I run these applications on my own time, with my own money, and my own resources.

Before I forget, thanks to all of the folks who came forward in the forums to explain what was actually happening and defend the work I’ve done. I’m tremendously flattered to receive that kind of support.