Moving SQS Messages To Another Queue

Recently we ran into a situation at work where we needed to move all messages from one of our dead letter queues, back onto the “live” queue for the service.

AWS doesn’t expose a way to move messages out of the box, so I wrote a small utility, which I creatively named sqsmv.

I wrote the project in Go because it is a great fit for command line programs, and I’ve also worked with the Go AWS SDK before.

I created a bunch of messages and tested it… First try! This is something I find happens more often when I’m building with Go, compared to doing the same task with a dynamic language.

sqsmv has worked well, successfully moving hundreds of messages between SQS queues. I hope someone out there finds it useful.

Processing Sidekiq jobs with Go Workers

I have a personal project where a Sidekiq worker (written in Ruby) was using between 150-200MB of RAM with 20 workers. These workers take images that have been uploaded to a Rails application, and upload them to an Amazon S3 bucket in the background. Most of the time these workers are idle. This project generates exactly $0.00 income, and I’m paying for that RAM, which even in the amazing days of cloud computing, is still too expensive to waste on a personal budget.

I decided to replace the Ruby process with a Go process in an attempt to reduce this wasted, idle RAM. (Sadly I do this kind of thing a lot).

To replace the Ruby process, I needed to write some Go code that could upload files to Amazon S3, and process Sidekiq jobs from a Redis queue.

The code that does the upload was really easy to write using the official AWS Go SDK library. I wrote nothing more than a little wrapper really.

I then added an application that picks up Sidekiq jobs from Redis. This wasn’t too difficult as the Gokiq library does most of the heavy lifting here. Working out a nice way to unpack the job data into my Worker struct without modifying the Gokiq library took some thought, but came out OK.

End Result

The Go process uses 28MB of RAM with 200 workers available. Compared to the 150-200MB of RAM with 20 workers that I had before, this is a nice little win.

Take a look at the s3uploader repository on Github if is sounds useful.

Multiple Go Versions

Running multiple versions of Go is simple. I see and hear chatter about it, and there are a few tools around to handle it for you, but it is so simple to do yourself that I’m surprised people reach for a tool to do this for them.

Mostly I run the latest stable version of Go, but for whatever reason (maybe I want to test out a release candidate), I do have multiple versions installed, and I don’t want multiple installations to clobber each other.

The following assumes you’re on some kind of *nix system, and you’re using Bash. Other OS’es and shells will be able to work as well with appropriate changes.

I like to keep my Go installations under ~/.go/versions. You can choose any location you like though.

mkdir -p ~/.go/versions

Let’s grab the latest version of Go (currently 1.6). I’m downloading the OS X archive here, but if you’re running another OS you would download the appropriate archive. Take a look at the Go Downloads page.

cd ~/.go/versions
tar -zxf go1.6.darwin-amd64.tar.gz
mv go go1.6
rm go1.6.darwin-amd64.tar.gz

Add the following environment variables to your ~/.bash_profile.

export GOPATH=~/p/go
export GOROOT=~/.go/versions/go1.6
export PATH=$GOROOT/bin:$GOPATH/bin:$PATH

This is where I set my default version of Go.

Note : GOPATH is where your Go projects will live, and GOROOT is where the Go installation lives. Don’t confuse the two.

Start a new shell and you should have go on your path.

$ go version
go version go1.6 darwin/amd64


So for whatever reason I also want an older version of Go installed.

cd ~/.go/versions
tar -zxf go1.5.3.darwin-amd64.tar.gz
mv go go1.5.3

Too easy. You must be getting bored.

Now I can temporarily switch to the another version by exporting the desired GOROOT and prefixing my current $PATH with $GOROOT/bin. In this case I am switching to the freshly installed GO 1.5.3.

$ export GOROOT=~/.go/versions/go1.5.3
$ export PATH=$GOROOT/bin:$PATH
$ go version
go version go1.5.3 darwin/amd64

Great Success!

About Scott Barr

I’m a software developer. I really like writing systems in Go (a.k.a Golang), Ruby and Python. I’m always working with HTML5 and Javascript, and I’ve even built a few native apps.

I’m the founder of the Brisbane Gophers, where people interested in Go get together and share what they’re learned with each other.

I often describe myself as a Digital Plumber, but I also play as a Data Scientist from time to time.

I’m always keen to partner on great ideas. Talk to me. I don’t bite.

Yes, I can and haved worked with “the cloud” :)

You can reach me on Twitter or take a look at some of my projects on Github.