Wednesday, February 26, 2014

Why I use just one GOPATH

I write programs in Go. All my Go projects have something in common: they can be classified by these categories:
  • personal
  • work
  • contributor/fork
And the following is true about all projects in each of these categories:
  • under source control
  • fully-qualified (unique) folder paths
The fully-qualified directory structure is important. I'll show you what I mean, with emphasis on the src folder:
  • ~
    • dev
      • src
        • github.com
          • mholt
            • papaparse
            • json-to-go
            • binding
            • makemypi
          • smartystreets
            • goconvey
          • codegangsta
            • martini
            • inject
          • martini-contrib
            • binding
        • bitbucket.org
          • mattholt
            • cs345
          • smartystreets
            • liveaddress
      • pkg
      • bin
      • etc
      • doc
Turns out that not only my Go projects can be categorized like this, but nearly everything I develop can be. This is why I've combined all my work, personal, and contributor/forked projects in one workspace, or GOPATH.

Benefits of using a single GOPATH

The structure above just happens to be how Go workspaces are defined (except the etc and doc folders; I use those for spike code and documentation). The package name is simply the path inside the src directory. Plus, this integrates seamlessly with go get.

I have also found that this structure allows me to combine my work, personal, and forked/contributor projects into ~/dev/src. One central place for my Go code means that I don't have to keep switching my GOPATH multiple times per day between work, school, and hobby/personal projects.

I actually used to maintain multiple GOPATHs and found myself getting confused about which ones had the latest dependencies. Updating GoConvey in one would cause me to forget that I hadn't in another, and when I wrote code in another for the latest GoConvey, it wouldn't work. That would stump me until I remembered that I have as many copies of GoConvey on my computer as I do GOPATHs. Besides juggling dependencies and compiled binaries and workspace configurations, always changing the environment variable was so annoying after a while.

For projects that aren't under source control, I still choose a folder path as if they would some day be committed to source control.

All dependencies are right in their place and I know where to find them. Since dependencies are also retrieved via source control, I can fork and branch the code to snapshot or modify it. Git becomes my dependency manager, and it's really intuitive and flexible in my experience.

Not only that, but binaries and object files are appropriately put into bin and pkg files, adjacent to the src folder. I added ~/dev/bin to my PATH and now all my compiled binaries are easily executable anywhere on my system.

A GOPATH for more than just Go code

Here's the really powerful, neat feature about this directory structure: I have since migrated all of my code into my GOPATH, even if it's not Go code. This same directory now houses all of my development projects. Can you believe I used to lose them sometimes? I would actually forget where I stored a certain project because they were scattered abroad in my dev folder, or websites in my sites folder... and it wasn't consistent.

Look again at my example above, and notice that I have Papa Parse and makemypi and json-to-go in my github.com folder, even though none of those are Go projects!

This way of using GOPATH is simple. I don't get lost. My code is nicely organized and tucked away into its proper place; the same place, in fact, that I find them on the Internet. Dependencies have never been a problem for me, and if they are, I can fork or branch the project thanks to git.

How do you organize your dev projects?

I have found this to be the most efficient, organized, simple, and consistent for my own setup, but everyone's situation is different. Do you structure your code like this? Would you try? Why or why not? (Let's not start a fight in the holy war of Go dependency management, though.)

Tuesday, February 4, 2014

Writing a Long-Poll Server in Go

This guide assumes you are already familiar with the Go HTTP server and channels.

Long-polling is a method to push server-side events down to one or more clients instantly. The client opens a request to the server, the server holds onto it until something happens, whereupon the server sends its response to the client. The client then re-issues a request and the process starts over.

Client

The Javascript code for the client, using jQuery, is rather simple:

var timeout = 60000; // milliseconds
var req = $.ajax({
  url: "/longpoll?timeout=" + timeout,
  timeout: timeout
}).done(pollSuccess).fail(pollFailed);

There are a few things to note:

  • The timeout variable is how long, in milliseconds, we want the server to hold onto our request before giving up.
  • We send the timeout value to the server for reasons that will be explained later.
  • pollSuccess will be executed when the server pushes its response down to you. It should first re-poll (which is asynchronous, so the rest of your callback will run immediately) and then process the server's response.
  • pollFailed will be executed if the request times out. jQuery will pass a "timeout" message to the function, so you can know that the server is still up, but merely that you need to re-poll. On the other hand, it may be an "error" in which case you should assume the server is down.
So you can see, the client is pretty straight-forward. Just don't forget to re-poll as soon as a response is received or the timeout occurs!

Server

Buckle up, we're about to make a channel of channels.

First, decide what type of data you want to send to your long-polling clients. We'll be using a string in these examples, but you may use a struct or any other data type. When you see chan chan string in this post, replace string with your own type.

Make a channel of channels of your data type and keep it handy:

lpchan := make(chan chan string)

Request Handler (Consumer)

When your server handles an incoming long-poll request, we need to make sure to hold onto it until we have something to push down to the client. But what if nothing happens for a while and we just keep holding onto it? That gets kind of bothersome, so we need to utilize that timeout value. Prepare for this by converting it to an integer:

timeout, err := strconv.Atoi(req.URL.Query().Get("timeout"))
if err != nil || timeout > 180000 || timeout < 0 {
  timeout = 60000  // default timeout is 60 seconds
}

Now we need to put this request into some sort of queue so we can respond to the client when something happens. Our outer channel conveniently works like a queue. We'll make a chan string to represent this particular request, and send it into the channel of channels. Since lpchan is unbuffered, it will block until something reads from it. This is where and how we "hold onto" the request, so we must mind the timeout:

myRequestChan = make(chan string)

select {
case lpchan <- myRequestChan:
case <-time.After(time.Duration(timeout) * time.Millisecond):
  return
}

The first case indicates that an event happened (we'll get to that in a moment). The second case means that the server has been idle, so we just drop the request because the client is dropping it and trying again anyway. (We don't want to hold onto a bunch of clients that aren't there anymore.) The very next thing we must do is try to read from the channel we just sent into the queue (lpchan).

response.Write([]byte(<-myRequestChan))

This blocks until it reads, then it sends the latest data to the client.

Application (Producer)

Now let's jump over to where "something happens" -- your Go application. When you're ready to tell the long-poll clients what's up, here's how:

Loop:
  for {
     select {
      case clientchan := <-lpchan:
        clientchan <- "hello, client!"
     default:
        break Loop
     }
  }

(You ever used a break-to-label before? Me neither!)

See how this works? We read from lpchan, our outer channel, until all the blocked requests were able to send. Those requests are sending channels that represent them. We treat those blocked sends as a queue.

We simply write the data to their channels until there are no more, then we're done.

As far as I can tell, that's the magic of a long-poll server in Go. It was easier than I thought. (I've probably overlooked some things, but so far, this has worked well, and it should ship with GoConvey's new web UI, fixing some of the known issues about auto-refresh.)