Tue, Sep 13, 2016
Recently we ran into a situation at work where we needed to move all
messages from one of our dead letter queues, back onto the “live”
queue for the service.
AWS doesn’t expose a way to move messages out of the box, so I wrote a
small utility, which I creatively
I wrote the project in Go because it is a great
fit for command line programs, and I’ve also worked with
the Go AWS SDK before.
I created a bunch of messages and tested it… First try! This is
something I find happens more often when I’m building with Go,
compared to doing the same task with a dynamic language.
sqsmv has worked well,
successfully moving hundreds of messages between SQS queues. I hope
someone out there finds it useful.
Tue, Aug 30, 2016
I have a personal project where a
Sidekiq worker (written in Ruby)
was using between 150-200MB of RAM with 20 workers. These workers take
images that have been uploaded to a Rails application, and upload them
to an Amazon S3 bucket in the background. Most of the time these
workers are idle. This project generates exactly $0.00 income, and I’m
paying for that RAM, which even in the amazing days of cloud
computing, is still too expensive to waste on a personal budget.
I decided to replace the Ruby process with a Go process in an attempt
to reduce this wasted, idle RAM. (Sadly I do this kind of thing a
To replace the Ruby process, I needed to write some Go code that could
upload files to Amazon S3, and process Sidekiq jobs from a Redis
The code that does the upload was really easy to write using the
official AWS Go SDK library. I
wrote nothing more than a little wrapper really.
I then added an application that picks up Sidekiq jobs from
Redis. This wasn’t too difficult as the
Gokiq library does most of the
heavy lifting here. Working out a nice way to unpack the job data into
my Worker struct without modifying the Gokiq library took some
thought, but came out OK.
The Go process uses 28MB of RAM with 200 workers available. Compared
to the 150-200MB of RAM with 20 workers that I had before, this is a
nice little win.
Take a look at the
s3uploader repository on
Github if is sounds useful.
Thu, Feb 25, 2016
Running multiple versions of Go is simple. I see and hear chatter
about it, and there are a few tools around to handle it for you, but
it is so simple to do yourself that I’m surprised people reach for a
tool to do this for them.
Mostly I run the latest stable version of Go, but for whatever reason
(maybe I want to test out a release candidate), I do have multiple
versions installed, and I don’t want multiple installations to clobber
The following assumes you’re on some kind of *nix system, and you’re
using Bash. Other OS’es and shells will be able to work as well with
I like to keep my Go installations under
~/.go/versions. You can
choose any location you like though.
mkdir -p ~/.go/versions
Let’s grab the latest version of Go (currently 1.6). I’m downloading
the OS X archive here, but if you’re running another OS you would
download the appropriate archive. Take a look at the Go
tar -zxf go1.6.darwin-amd64.tar.gz
mv go go1.6
Add the following environment variables to your
This is where I set my default version of Go.
GOPATH is where your Go projects will live, and
is where the Go installation lives. Don’t confuse the two.
Start a new shell and you should have
go on your path.
$ go version
go version go1.6 darwin/amd64
So for whatever reason I also want an older version of Go installed.
tar -zxf go1.5.3.darwin-amd64.tar.gz
mv go go1.5.3
Too easy. You must be getting bored.
Now I can temporarily switch to the another version by exporting the
GOROOT and prefixing my current
$GOROOT/bin. In this case I am switching to the freshly installed GO 1.5.3.
$ export GOROOT=~/.go/versions/go1.5.3
$ export PATH=$GOROOT/bin:$PATH
$ go version
go version go1.5.3 darwin/amd64
Sun, Feb 14, 2016
I’m a software developer. I really like writing systems in Go (a.k.a Golang),
built a few native apps.
I’m the founder of the
Brisbane Gophers, where
people interested in Go get together and share what they’re learned with each
I often describe myself as a Digital Plumber, but I also play as a
Data Scientist from time to time.
I’m always keen to partner on great ideas. Talk to me. I don’t bite.
Yes, I can and haved worked with “the cloud” :)
You can reach me on Twitter or take a look
at some of my projects on Github.