major.io words of wisdom from a systems engineer

Reincarnation of Twitter’s realtime XMPP search term tracking with ruby

When Twitter was still in its early stages, you could track certain search terms in near-realtime via Jabber. It was quite popular and its performance degraded over time as more users signed up and began posting updates. Eventually, Twitter killed the jabber bot altogether. Many users have asked when it will return.

Well, it hasn’t returned, but you can build your own replacement with ruby, a jabber account, and a few gems. While it won’t do everything that the original jabber bot did, you can still track tweets mentioning certain terms very quickly.

Here’s how to get started:

First, install the tweetstream and xmpp4r-simple gems:

gem install tweetstream xmpp4r-simple

Next, you’ll need a jabber account. You’ll probably want to make one for the exclusive use of your jabber bot. I chose to make up a quick account at ChatMask for mine.

The last step is to drop a copy of this script on your server:

#!/usr/bin/ruby
require 'rubygems'
require 'tweetstream/client'
require 'tweetstream/hash'
require 'tweetstream/status'
require 'tweetstream/user'
require 'tweetstream/daemon'
require 'xmpp4r-simple'

jabber = Jabber::Simple.new('[email protected]','jabberpassword')

tweets = TweetStream::Client.new(twitterusername,twitterpassword)

tweets.track('celtics','lakers','finals','nba') do |status, client|
  imtext = "#{status.user.screen_name}: #{status.text} \r\n"
  imtext += "[http://twitter.com/#{status.user.screen_name}/status/#{status.id}]"
  jabber.deliver("[email protected]",imtext)
end

jabber.disconnect

You’ll want to be sure to fill in the following:

  • your jabber bot’s username and password
  • the username and password for the twitter account that will monitor the stream
  • the search terms you want to track
  • the destination jabber account where the messages should be sent

Ensure that your jabber account has authorized the jabber bot’s account so that you’ll actually receive the messages. Also, Twitter is very strict with their streaming API tracking terms. It’s a good idea to review their entire Streaming API documentation to ensure that you’re not going to end up having your account temporarily or permanently blacklisted.

Once everything is ready to go, you can just run the script within GNU screen or via nohup. There’s still a bit more error checking to do around jabber reconnections, but the script has run non-stop for well over two weeks at a time without a failure.