Tagged with python …

Mar 16 2013
Leave a comment

New icanhaz features: reverse DNS and traceroutes

After adding some upgrades for icanhazip.com, I wanted to go a bit further. Adding reverse DNS (PTR) lookups and traceroutes seemed like a decent idea!

Want to beta test some new features on icanhazptr.com and icanhaztrace.com? Give them a try!

Getting your reverse DNS entry is easy:

$ curl -4 icanhazptr.com
ord.icanhazip.com
$ curl -6 icanhazptr.com
ord.icanhazip.com

Traceroutes are straightforward as well:

$ curl -4 icanhaztrace.com
traceroute to 166.78.118.193 (166.78.118.193), 30 hops max, 60 byte packets
 1  212.111.33.229  20.031 ms
 2  212.111.33.233  1.011 ms
 3  149.11.30.61  107.976 ms
...
$ curl -6 icanhaztrace.com
traceroute to 2001:4801:7818:6:abc5:ba2c:ff10:275f (2001:4801:7818:6:abc5:ba2c:ff10:275f), 30 hops max, 80 byte packets
 1  2a01:7e00:ffff:0:8a43:e1ff:fea3:fa7f  2.183 ms
 2  2001:4d78:fe01:2:1:3:b90:1  1.330 ms
 3  2001:978:2:45::d:1  8.388 ms
...

While this sits in beta, here are some things to keep in mind:

  • If a PTR record doesn’t exist for your IP address, your IP address will be returned
  • Failing traceroutes will cause your IP address to be returned
  • A PTR record will be chosen at random if multiple PTR records are returned
  • PTR lookups for traceroutes are currently disabled

Let me know if you find any bugs.

Tagged , , , , , ,
Nov 25 2012
Leave a comment

Relocating a python virtual environment

Python’s virtual environment capability is extremely handy for situations where you don’t want the required modules for a particular python project to get mixed up with your system-wide installed modules. If you work on large python projects (like OpenStack), you’ll find that the applications may require certain versions of python modules to operate properly. If these versions differ from the system-wide python modules you already have installed, you might get unexpected results when you try to run the unit tests.

If you build a virtual environment and inspect the files found within the bin directory of the virtual environment, you’ll find that the first line in the executable scripts is set to use the python version specific to that virtual environment. Here’s an example from a virtual environment containing the OpenStack glance project:

#!/home/major/glance/.venv/bin/python
# EASY-INSTALL-SCRIPT: 'glance==2013.1','glance-api'
__requires__ = 'glance==2013.1'
import pkg_resources
pkg_resources.run_script('glance==2013.1', 'glance-api')

However, what if I wanted to take this virtual environment and place it somewhere else on the server where multiple people could use it? The path in the first line of the scripts in bin will surely break.

The first option is to make the virtual environment relocatable. This can produce unexpected results for some software projects, so be sure to test it out before trying to use it in a production environment.

$ virtualenv --relocatable .venv

A quick check of the same python file now shows this:

#!/usr/bin/env python2.6
 
import os; activate_this=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'activate_this.py'); execfile(activate_this, dict(__file__=activate_this)); del os, activate_this
 
# EASY-INSTALL-SCRIPT: 'glance==2013.1','glance-api'

This allows for the path to the activate_this.py script to be determined at runtime and allows you to move your virtual environment wherever you like.

In situations where one script within bin would import another script within bin, things can get a little dicey. These are edge cases, of course, but you can get a similar effect by adjusting the path in the first line of each file within bin to the new location of the virtual environment. If you move the virtual environment again, be sure to alter the paths again with sed.

Tagged , ,
Jun 08 2012
Leave a comment

Keep tabs on OpenStack development with OpenStack Watch on Twitter

It’s no secret that I’m a fan of Twitter and OpenStack. I found myself needing a better way to follow the rapid pace of OpenStack development and I figured that a Twitter bot would be a pretty good method for staying up to date.

I’d like to invite you to check out @openstackwatch.

First things first, it’s a completely unofficial project that I worked on during my spare time and it’s not affiliated with OpenStack in any way. If it breaks, it’s most likely my fault.

The bot watches for ticket status changes in OpenStack’s Gerrit server and makes a tweet about the change within a few minutes. Every tweet contains the commit’s project, owner, status, and a brief summary of the change. In addition, you’ll get a link directly to the review page on the Gerrit server. Here’s an example:

Hey! It's Dan!

If you’re not a fan of Twitter, there’s a link to the RSS feed in the bio section, or you can just add this URL to your RSS feed reader:

If you can come up with any ideas for improvements, please let me know!

Tagged , , ,
Jun 05 2012
3 Comments

supernova: Manage multiple OpenStack nova environments with ease

While working on multiple nova (OpenStack Compute) environments at Rackspace, I found myself thrashing between multiple terminal windows where I had exported environment variables for python-novaclient to use. I ended up requesting some image and instance deletions in a terminal window only to find that I’d done the deletions in the wrong nova environment. Once I realized what I’d done (and after a small bit of cursing), I knew there had to be a better way to work with multiple environments.

That’s the purpose behind a small python project of mine: supernova.

Using supernova gives you a nice set of benefits:

  • switch between environments quickly
  • no worrying about which environment variables are exported in which terminal
  • novarc files are a thing of the past
  • share your simple configuration file skeleton with your teams
  • credentials can be stored in your OS keyring/keychain
  • add novaclient debugging to particular requests without touching configuration files

Installation is very straightforward:

git clone git://github.com/rackerhacker/supernova.git
cd supernova
python setup.py install

All of the configuration instructions and usage examples are over in GitHub. As with any of the code I write, if you find a problem or spot an idea for an improvement, submit an issue or pull request. I try to jump on those as soon as I can.

Tagged , , , ,
Mar 28 2012
Leave a comment

mysql-json-bridge: a simple JSON API for MySQL

My quest to get better at Python led me to create a new project on GitHub. It’s called mysql-json-bridge and it’s ready for you to use.

Why do we need a JSON API for MySQL?
The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That’s where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?
It all starts with a simple HTTP POST. I’ve become a big fan of the Python requests module. If you’re using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they’re converted to JSON and returned via the same HTTP connection.

What technology does it use?
Flask does the heavy lifting for the HTTP requests and Facebook’s Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?
Yes. I haven’t done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You’re terrible at Python.
All feedback (and every pull request) is welcome. I’m still getting the hang of Python (hey, I’ve only been writing in it seriously for a few weeks!) and I’m always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.

Tagged , , , , , ,
← Older posts