The Go Blog

Deploying Go servers with Docker

26 September 2014

Introduction

This week Docker announced official base images for Go and other major languages, giving programmers a trusted and easy way to build containers for their Go programs.

In this article we'll walk through a recipe for creating a Docker container for a simple Go web application and deploying that container to Google Compute Engine. If you're not familiar with Docker, you should read Understanding Docker before reading on.

The demo app

For our demonstration we will use the outyet program from the Go examples repository, a simple web server that reports whether the next version of Go has been released (designed to power sites like isgo1point4.outyet.org). It has no dependencies outside the standard library and requires no additional data files at run time; for a web server, it's about as simple as it gets.

Use "go get" to fetch and install outyet in your workspace:

$ go get github.com/golang/example/outyet

Write a Dockerfile

Create a file named Dockerfile in the outyet directory with the following contents:

# Start from a Debian image with the latest version of Go installed
# and a workspace (GOPATH) configured at /go.
FROM golang

# Copy the local package files to the container's workspace.
ADD . /go/src/github.com/golang/example/outyet

# Build the outyet command inside the container.
# (You may fetch or manage dependencies here,
# either manually or with a tool like "godep".)
RUN go install github.com/golang/example/outyet

# Run the outyet command by default when the container starts.
ENTRYPOINT /go/bin/outyet

# Document that the service listens on port 8080.
EXPOSE 8080

This Dockerfile specifies how to construct a container that runs outyet, starting with the basic dependencies (a Debian system with Go installed; the official golang docker image), adding the outyet package source, building it, and then finally running it.

The ADD, RUN, and ENTRYPOINT steps are common tasks for any Go project. To simplify this, there is an onbuild variant of the golang image that automatically copies the package source, fetches the application dependencies, builds the program, and configures it to run on startup.

With the onbuild variant, the Dockerfile is much simpler:

FROM golang:onbuild
EXPOSE 8080

Build and run the image

Invoke Docker from the package directory to build an image using the Dockerfile:

$ docker build -t outyet .

This will fetch the golang base image from Docker Hub, copy the package source to it, build the package inside it, and tag the resulting image as outyet.

To run a container from the resulting image:

$ docker run --publish 6060:8080 --name test --rm outyet

The --publish flag tells docker to publish the container's port 8080 on the external port 6060.

The --name flag gives our container a predictable name to make it easier to work with.

The --rm flag tells docker to remove the container image when the outyet server exits.

With the container running, open http://localhost:6060/ in a web browser and you should see something like this:

(If your docker daemon is running on another machine (or in a virtual machine), you should replace localhost with the address of that machine. If you're using boot2docker on OS X or Windows you can find that address with boot2docker ip.)

Now that we've verified that the image works, shut down the running container from another terminal window:

$ docker stop test

Create a repository on Docker Hub

Docker Hub, the container registry from which we pulled the golang image earlier, offers a feature called Automated Builds that builds images from a GitHub or BitBucket repository.

By committing the Dockerfile to the repository and creating an automated build for it, anyone with Docker installed can download and run our image with a single command. (We will see the utility of this in the next section.)

To set up an Automated Build, commit the Dockerfile to your repo on GitHub or BitBucket, create an account on Docker Hub, and follow the instructions for creating an Automated Build.

When you're done, you can run your container using the name of the automated build:

$ docker run goexample/outyet

(Replace goexample/outyet with the name of the automated build you created.)

Deploy the container to Google Compute Engine

Google provides container-optimized Google Compute Engine images that make it easy to spin up a virtual machine running an arbitrary Docker container. On startup, a program running on the instance reads a configuration file that specifies which container to run, fetches the container image, and runs it.

Create a containers.yaml file that specifies the docker image to run and the ports to expose:

version: v1beta2
containers:
- name: outyet
  image: goexample/outyet
  ports:
  - name: http
    hostPort: 80
    containerPort: 8080

(Note that we're publishing the container's port 8080 as external port 80, the default port for serving HTTP traffic. And, again, you should replace goexample/outyet with the name of your Automated Build.)

Use the gcloud tool to create a VM instance running the container:

$ gcloud compute instances create outyet \
    --image container-vm-v20140925 \
    --image-project google-containers \
    --metadata-from-file google-container-manifest=containers.yaml \
    --tags http-server \
    --zone us-central1-a \
    --machine-type f1-micro

The first argument (outyet) specifies the instance name, a convenient label for administrative purposes.

The --image and --image-project flags specify the special container-optimized system image to use (copy these flags verbatim).

The --metadata-from-file flag supplies your containers.yaml file to the VM.

The --tags flag tags your VM instance as an HTTP server, adjusting the firewall to expose port 80 on the public network interface.

The --zone and --machine-type flags specify the zone in which to run the VM and the type of machine to run. (To see a list of machine types and the zones, run gcloud compute machine-types list.)

Once this has completed, the gcloud command should print some information about the instance. In the output, locate the networkInterfaces section to find the instance's external IP address. Within a couple of minutes you should be able to access that IP with your web browser and see the "Has Go 1.4 been released yet?" page.

(To see what's happening on the new VM instance you can ssh into it with gcloud compute ssh outyet. From there, try sudo docker ps to see which Docker containers are running.)

Learn more

This is just the tip of the iceberg—there's a lot more you can do with Go, Docker, and Google Compute Engine.

To learn more about Docker, see their extensive documentation.

To learn more about Docker and Go, see the official golang Docker Hub repository and Kelsey Hightower's Optimizing Docker Images for Static Go Binaries.

To learn more about Docker and Google Compute Engine, see the Container-optimized VMs page and the google/docker-registry Docker Hub repository.

By Andrew Gerrand

Constants

25 August 2014

Introduction

Go is a statically typed language that does not permit operations that mix numeric types. You can't add a float64 to an int, or even an int32 to an int. Yet it is legal to write 1e6*time.Second or math.Exp(1) or even 1<<('\t'+2.0). In Go, constants, unlike variables, behave pretty much like regular numbers. This post explains why that is and what it means.

Background: C

In the early days of thinking about Go, we talked about a number of problems caused by the way C and its descendants let you mix and match numeric types. Many mysterious bugs, crashes, and portability problems are caused by expressions that combine integers of different sizes and "signedness". Although to a seasoned C programmer the result of a calculation like

unsigned int u = 1e9;
long signed int i = -1;
... i + u ...

may be familiar, it isn't a priori obvious. How big is the result? What is its value? Is it signed or unsigned?

Nasty bugs lurk here.

C has a set of rules called "the usual arithmetic conversions" and it is an indicator of their subtlety that they have changed over the years (introducing yet more bugs, retroactively).

When designing Go, we decided to avoid this minefield by mandating that there is no mixing of numeric types. If you want to add i and u, you must be explicit about what you want the result to be. Given

var u uint
var i int

you can write either uint(i)+u or i+int(u), with both the meaning and type of the addition clearly expressed, but unlike in C you cannot write i+u. You can't even mix int and int32, even when int is a 32-bit type.

This strictness eliminates a common cause of bugs and other failures. It is a vital property of Go. But it has a cost: it sometimes requires programmers to decorate their code with clumsy numeric conversions to express their meaning clearly.

And what about constants? Given the declarations above, what would make it legal to write i = 0 or u = 0? What is the type of 0? It would be unreasonable to require constants to have type conversions in simple contexts such as i = int(0).

We soon realized the answer lay in making numeric constants work differently from how they behave in other C-like languages. After much thinking and experimentation, we came up with a design that we believe feels right almost always, freeing the programmer from converting constants all the time yet being able to write things like math.Sqrt(2) without being chided by the compiler.

In short, constants in Go just work, most of the time anyway. Let's see how that happens.

Terminology

First, a quick definition. In Go, const is a keyword introducing a name for a scalar value such as 2 or 3.14159 or "scrumptious". Such values, named or otherwise, are called constants in Go. Constants can also be created by expressions built from constants, such as 2+3 or 2+3i or math.Pi/2 or ("go"+"pher").

Some languages don't have constants, and others have a more general definition of constant or application of the word const. In C and C++, for instance, const is a type qualifier that can codify more intricate properties of more intricate values.

But in Go, a constant is just a simple, unchanging value, and from here on we're talking only about Go.

String constants

There are many kinds of numeric constants—integers, floats, runes, signed, unsigned, imaginary, complex—so let's start with a simpler form of constant: strings. String constants are easy to understand and provide a smaller space in which to explore the type issues of constants in Go.

A string constant encloses some text between double quotes. (Go has also has raw string literals, enclosed by backquotes ``, but for the purpose of this discussion they have all the same properties.) Here is a string constant:

"Hello, 世界"

(For much more detail about the representation and interpretation of strings, see this blog post.)

What type does this string constant have? The obvious answer is string, but that is wrong.

This is an untyped string constant, which is to say it is a constant textual value that does not yet have a fixed type. Yes, it's a string, but it's not a Go value of type string. It remains an untyped string constant even when given a name:

const hello = "Hello, 世界"

After this declaration, hello is also an untyped string constant. An untyped constant is just a value, one not yet given a defined type that would force it to obey the strict rules that prevent combining differently typed values.

It is this notion of an untyped constant that makes it possible for us to use constants in Go with great freedom.

So what, then, is a typed string constant? It's one that's been given a type, like this:

const typedHello string = "Hello, 世界"

Notice that the declaration of typedHello has an explicit string type before the equals sign. This means that typedHello has Go type string, and cannot be assigned to a Go variable of a different type. That is to say, this code works:

// +build OMIT

package main

import "fmt"

const typedHello string = "Hello, 世界"

func main() {
    var s string
    s = typedHello
    fmt.Println(s)
}

but this does not:

// +build OMIT

package main

import "fmt"

const typedHello string = "Hello, 世界"

func main() {
    type MyString string
    var m MyString
    m = typedHello // Type error
    fmt.Println(m)
}

The variable m has type MyString and cannot be assigned a value of a different type. It can only be assigned values of type MyString, like this:

// +build OMIT

package main

import "fmt"

const typedHello string = "Hello, 世界"

func main() {
	type MyString string
	var m MyString
    const myStringHello MyString = "Hello, 世界"
    m = myStringHello // OK
    fmt.Println(m)
}

or by forcing the issue with a conversion, like this:

// +build OMIT

package main

import "fmt"

const typedHello string = "Hello, 世界"

func main() {
	type MyString string
	var m MyString
    m = MyString(typedHello)
    fmt.Println(m)
}

Returning to our untyped string constant, it has the helpful property that, since it has no type, assigning it to a typed variable does not cause a type error. That is, we can write

m = "Hello, 世界"

or

m = hello

because, unlike the typed constants typedHello and myStringHello, the untyped constants "Hello, 世界" and hello have no type. Assigning them to a variable of any type compatible with strings works without error.

These untyped string constants are strings, of course, so they can only be used where a string is allowed, but they do not have type string.

Default type

As a Go programmer, you have certainly seen many declarations like

str := "Hello, 世界"

and by now you might be asking, "if the constant is untyped, how does str get a type in this variable declaration?" The answer is that an untyped constant has a default type, an implicit type that it transfers to a value if a type is needed where none is provided. For untyped string constants, that default type is obviously string, so

str := "Hello, 世界"

or

var str = "Hello, 世界"

means exactly the same as

var str string = "Hello, 世界"

One way to think about untyped constants is that they live in a kind of ideal space of values, a space less restrictive than Go's full type system. But to do anything with them, we need to assign them to variables, and when that happens the variable (not the constant itself) needs a type, and the constant can tell the variable what type it should have. In this example, str becomes a value of type string because the untyped string constant gives the declaration its default type, string.

In such a declaration, a variable is declared with a type and initial value. Sometimes when we use a constant, however, the destination of the value is not so clear. For instance consider this statement:

// +build OMIT

package main

import "fmt"

func main() {
    fmt.Printf("%s", "Hello, 世界")
}

The signature of fmt.Printf is

func Printf(format string, a ...interface{}) (n int, err error)

which is to say its arguments (after the format string) are interface values. What happens when fmt.Printf is called with an untyped constant is that an interface value is created to pass as an argument, and the concrete type stored for that argument is the default type of the constant. This process is analogous to what we saw earlier when declaring an initialized value using an untyped string constant.

You can see the result in this example, which uses the format %v to print the value and %T to print the type of the value being passed to fmt.Printf:

// +build OMIT

package main

import "fmt"

const hello = "Hello, 世界"

func main() {
    fmt.Printf("%T: %v\n", "Hello, 世界", "Hello, 世界")
    fmt.Printf("%T: %v\n", hello, hello)
}

If the constant has a type, that goes into the interface, as this example shows:

// +build OMIT

package main

import "fmt"

type MyString string

const myStringHello MyString = "Hello, 世界"

func main() {
    fmt.Printf("%T: %v\n", myStringHello, myStringHello)
}

(For more information about how interface values work, see the first sections of this blog post.)

In summary, a typed constant obeys all the rules of typed values in Go. On the other hand, an untyped constant does not carry a Go type in the same way and can be mixed and matched more freely. It does, however, have a default type that is exposed when, and only when, no other type information is available.

Default type determined by syntax

The default type of an untyped constant is determined by its syntax. For string constants, the only possible implicit type is string. For numeric constants, the implicit type has more variety. Integer constants default to int, floating-point constants float64, rune constants to rune (an alias for int32), and imaginary constants to complex128. Here's our canonical print statement used repeatedly to show the default types in action:

// +build OMIT

package main

import "fmt"

func main() {
    fmt.Printf("%T %v\n", 0, 0)
    fmt.Printf("%T %v\n", 0.0, 0.0)
    fmt.Printf("%T %v\n", 'x', 'x')
    fmt.Printf("%T %v\n", 0i, 0i)
}

(Exercise: Explain the result for 'x'.)

Booleans

Everything we said about untyped string constants can be said for untyped boolean constants. The values true and false are untyped boolean constants that can be assigned to any boolean variable, but once given a type, boolean variables cannot be mixed:

// +build OMIT

package main

import "fmt"

func main() {
    type MyBool bool
    const True = true
    const TypedTrue bool = true
    var mb MyBool
    mb = true      // OK
    mb = True      // OK
    mb = TypedTrue // Bad
    fmt.Println(mb)
}

Run the example and see what happens, then comment out the "Bad" line and run it again. The pattern here follows exactly that of string constants.

Floats

Floating-point constants are just like boolean constants in most respects. Our standard example works as expected in translation:

// +build OMIT

package main

import "fmt"

func main() {
    type MyFloat64 float64
    const Zero = 0.0
    const TypedZero float64 = 0.0
    var mf MyFloat64
    mf = 0.0       // OK
    mf = Zero      // OK
    mf = TypedZero // Bad
    fmt.Println(mf)
}

One wrinkle is that there are two floating-point types in Go: float32 and float64. The default type for a floating-point constant is float64, although an untyped floating-point constant can be assigned to a float32 value just fine:

// +build OMIT

package main

import "fmt"

func main() {
	const Zero = 0.0
	const TypedZero float64 = 0.0
    var f32 float32
    f32 = 0.0
    f32 = Zero      // OK: Zero is untyped
    f32 = TypedZero // Bad: TypedZero is float64 not float32.
    fmt.Println(f32)
}

Floating-point values are a good place to introduce the concept of overflow, or the range of values.

Numeric constants live in an arbitrary-precision numeric space; they are just regular numbers. But when they are assigned to a variable the value must be able to fit in the destination. We can declare a constant with a very large value:

    const Huge = 1e1000

—that's just a number, after all—but we can't assign it or even print it. This statement won't even compile:

// +build OMIT

package main

import "fmt"

func main() {
	const Huge = 1e1000
	// START OMIT
    fmt.Println(Huge)
	// STOP OMIT
}

The error is, "constant 1.00000e+1000 overflows float64", which is true. But Huge might be useful: we can use it in expressions with other constants and use the value of those expressions if the result can be represented in the range of a float64. The statement,

// +build OMIT

package main

import "fmt"

func main() {
	const Huge = 1e1000
	// START OMIT
    fmt.Println(Huge / 1e999)
	// STOP OMIT
}

prints 10, as one would expect.

In a related way, floating-point constants may have very high precision, so that arithmetic involving them is more accurate. The constants defined in the math package are given with many more digits than are available in a float64. Here is the definition of math.Pi:

Pi    = 3.14159265358979323846264338327950288419716939937510582097494459

When that value is assigned to a variable, some of the precision will be lost; the assignment will create the float64 (or float32) value closest to the high-precision value. This snippet

// +build OMIT

package main

import (
	"fmt"
	"math"
)

func main() {
    pi := math.Pi
    fmt.Println(pi)
}

prints 3.141592653589793.

Having so many digits available means that calculations like Pi/2 or other more intricate evaluations can carry more precision until the result is assigned, making calculations involving constants easier to write without losing precision. It also means that there is no occasion in which the floating-point corner cases like infinities, soft underflows, and NaNs arise in constant expressions. (Division by a constant zero is a compile-time error, and when everything is a number there's no such thing as "not a number".)

Complex numbers

Complex constants behave a lot like floating-point constants. Here's a version of our now-familiar litany translated into complex numbers:

// +build OMIT

package main

import "fmt"

func main() {
    type MyComplex128 complex128
    const I = (0.0 + 1.0i)
    const TypedI complex128 = (0.0 + 1.0i)
    var mc MyComplex128
    mc = (0.0 + 1.0i) // OK
    mc = I            // OK
    mc = TypedI       // Bad
    fmt.Println(mc)
}

The default type of a complex number is complex128, the larger-precision version composed of two float64 values.

For clarity in our example, we wrote out the full expression (0.0+1.0i), but this value can be shortened to 0.0+1.0i, 1.0i or even 1i.

Let's play a trick. We know that in Go, a numeric constant is just a number. What if that number is a complex number with no imaginary part, that is, a real? Here's one:

    const Two = 2.0 + 0i

That's an untyped complex constant. Even though it has no imaginary part, the syntax of the expression defines it to have default type complex128. Therefore, if we use it to declare a variable, the default type will be complex128. The snippet

// +build OMIT

package main

import "fmt"

func main() {
	const Two = 2.0 + 0i
    s := Two
    fmt.Printf("%T: %v\n", s, s)
}

prints complex128: (2+0i). But numerically, Two can be stored in a scalar floating-point number, a float64 or float32, with no loss of information. Thus we can assign Two to a float64, either in an initialization or an assignment, without problems:

// +build OMIT

package main

import "fmt"

func main() {
	const Two = 2.0 + 0i
    var f float64
    var g float64 = Two
    f = Two
    fmt.Println(f, "and", g)
}

The output is 2 and 2. Even though Two is a complex constant, it can be assigned to scalar floating-point variables. This ability for a constant to "cross" types like this will prove useful.

Integers

At last we come to integers. They have more moving parts—many sizes, signed or unsigned, and more—but they play by the same rules. For the last time, here is our familiar example, using just int this time:

// +build OMIT

package main

import "fmt"

func main() {
    type MyInt int
    const Three = 3
    const TypedThree int = 3
    var mi MyInt
    mi = 3          // OK
    mi = Three      // OK
    mi = TypedThree // Bad
    fmt.Println(mi)
}

The same example could be built for any of the integer types, which are:

int int8 int16 int32 int64
uint uint8 uint16 uint32 uint64
uintptr

(plus the aliases byte for uint8 and rune for int32). That's a lot, but the pattern in the way constants work should be familiar enough by now that you can see how things will play out.

As mentioned above, integers come in a couple of forms and each form has its own default type: int for simple constants like 123 or 0xFF or -14 and rune for quoted characters like 'a', '世' or '\r'.

No constant form has as its default type an unsigned integer type. However, the flexibility of untyped constants means we can initialize unsigned integer variables using simple constants as long as we are clear about the type. It's analogous to how we can initialize a float64 using a complex number with zero imaginary part. Here are several different ways to initialize a uint; all are equivalent, but all must mention the type explicitly for the result to be unsigned.

var u uint = 17
var u = uint(17)
u := uint(17)

Similarly to the range issue mentioned in the section on floating-point values, not all integer values can fit in all integer types. There are two problems that might arise: the value might be too large, or it might be a negative value being assigned to an unsigned integer type. For instance, int8 has range -128 through 127, so constants outside of that range can never be assigned to a variable of type int8:

// +build OMIT

package main

func main() {
	// START OMIT
    var i8 int8 = 128 // Error: too large.
	// STOP OMIT
	_ = i8
}

Similarly, uint8, also known as byte, has range 0 through 255, so a large or negative constant cannot be assigned to a uint8:

// +build OMIT

package main

func main() {
	// START OMIT
    var u8 uint8 = -1 // Error: negative value.
	// STOP OMIT
	_ = u8
}

This type-checking can catch mistakes like this one:

// +build OMIT

package main

func main() {
    type Char byte
    var c Char = '世' // Error: '世' has value 0x4e16, too large.
	_ = c
}

If the compiler complains about your use of a constant, it's likely a real bug like this.

An exercise: The largest unsigned int

Here is an informative little exercise. How do we express a constant representing the largest value that fits in a uint? If we were talking about uint32 rather than uint, we could write

const MaxUint32 = 1<<32 - 1

but we want uint, not uint32. The int and uint types have equal unspecified numbers of bits, either 32 or 64. Since the number of bits available depends on the architecture, we can't just write down a single value.

Fans of two's-complement arithmetic, which Go's integers are defined to use, know that the representation of -1 has all its bits set to 1, so the bit pattern of -1 is internally the same as that of the largest unsigned integer. We therefore might think we could write

// +build OMIT

package main

func main() {
	// START OMIT
    const MaxUint uint = -1 // Error: negative value
	// STOP OMIT
}

but that is illegal because -1 cannot be represented by an unsigned variable; -1 is not in the range of unsigned values. A conversion won't help either, for the same reason:

// +build OMIT

package main

func main() {
	// START OMIT
    const MaxUint uint = uint(-1) // Error: negative value
	// STOP OMIT
}

Even though at run-time a value of -1 can be converted to an unsigned integer, the rules for constant conversions forbid this kind of coercion at compile time. That is to say, this works:

// +build OMIT

package main

func main() {
    var u uint
    var v = -1
    u = uint(v)
	_ = u
}

but only because v is a variable; if we made v a constant, even an untyped constant, we'd be back in forbidden territory:

// +build OMIT

package main

func main() {
    var u uint
    const v = -1
    u = uint(v) // Error: negative value
	_ = u
}

We return to our previous approach, but instead of -1 we try ^0, the bitwise negation of an arbitrary number of zero bits. But that fails too, for a similar reason: In the space of numeric values, ^0 represents an infinite number of ones, so we lose information if we assign that to any fixed-size integer:

// +build OMIT

package main

func main() {
	// START OMIT
    const MaxUint uint = ^0 // Error: overflow
	// STOP OMIT
}

How then do we represent the largest unsigned integer as a constant?

The key is to constrain the operation to the number of bits in a uint and avoiding values, such as negative numbers, that are not representable in a uint. The simplest uint value is the typed constant uint(0). If uints have 32 or 64 bits, uint(0) has 32 or 64 zero bits accordingly. If we invert each of those bits, we'll get the correct number of one bits, which is the largest uint value.

Therefore we don't flip the bits of the untyped constant 0, we flip the bits of the typed constant uint(0). Here, then, is our constant:

// +build OMIT

package main

import "fmt"

func main() {
    const MaxUint = ^uint(0)
    fmt.Printf("%x\n", MaxUint)
}

Whatever the number of bits it takes to represent a uint in the current execution environment (on the playground, it's 32), this constant correctly represents the largest value a variable of type uint can hold.

If you understand the analysis that got us to this result, you understand all the important points about constants in Go.

Numbers

The concept of untyped constants in Go means that all the numeric constants, whether integer, floating-point, complex, or even character values, live in a kind of unified space. It's when we bring them to the computational world of variables, assignments, and operations that the actual types matter. But as long as we stay in the world of numeric constants, we can mix and match values as we like. All these constants have numeric value 1:

1
1.000
1e3-99.0*10-9
'\x01'
'\u0001'
'b' - 'a'
1.0+3i-3.0i

Therefore, although they have different implicit default types, written as untyped constants they can be assigned to a variable of any integer type:

// +build OMIT

package main

import "fmt"

func main() {
    var f float32 = 1
    var i int = 1.000
    var u uint32 = 1e3 - 99.0*10.0 - 9
    var c float64 = '\x01'
    var p uintptr = '\u0001'
    var r complex64 = 'b' - 'a'
    var b byte = 1.0 + 3i - 3.0i

    fmt.Println(f, i, u, c, p, r, b)
}

The output from this snippet is: 1 1 1 1 1 (1+0i) 1.

You can even do nutty stuff like

// +build OMIT

package main

import "fmt"

func main() {
    var f = 'a' * 1.5
    fmt.Println(f)
}

which yields 145.5, which is pointless except to prove a point.

But the real point of these rules is flexibility. That flexibility means that, despite the fact that in Go it is illegal in the same expression to mix floating-point and integer variables, or even int and int32 variables, it is fine to write

sqrt2 := math.Sqrt(2)

or

const millisecond = time.Second/1e3

or

bigBufferWithHeader := make([]byte, 512+1e6)

and have the results mean what you expect.

Because in Go, numeric constants work as you expect: like numbers.

By Rob Pike

Go at OSCON

20 August 2014

Introduction

What happens in Portland in July? OSCON! At this year's conference, Go was more present than ever before, with five talks, two workshops, a Birds of a Feather session, and a meetup.

Talks

Matt Stine talked about his experience switching from Java to Go with A recovering Java developer learns Go while Steve Francia presented Painless Data Storage with MongoDB and Go. Steve also presented Go for Object Oriented Programmers, where he explained how some object oriented concepts can be implemented in Go.

Finally, Josh Bleecher Snyder talked about his experience writing tools to work with Go source code in Gophers with hammers, and Francesc Campoy talked about all the things that could have gone wrong and what the Go team did to prevent them Inside the Go playground.

Workshops

At the beginning of OSCON's workshop day, Steve Francia presented how to build a web application and a CLI tool during Getting started with Go to a big room full of Gophers.

In the afternoon, Chris McEniry gave his Quick introduction to system tools programming with Go where he went over some useful skills to write system tools using Go and its standard library.

Additional events

To take advantage of the increased Gopher population in Portland during OSCON, we organized two extra events: the first PDXGolang meetup and a Birds of a Feather session.

At the meetup Francesc Campoy talked about Go Best Practices and Kelsey Hightower gave a great introduction to Kubernetes, a container management system for clusters written in Go by Google. If you live in Portland, make sure you join the group and come along to the next meeting.

The "Birds of a Feather" (or, more aptly, "Gophers of a Feather") was a lot of fun for everyone involved. We hope to see more of you there next year.

In conclusion

Thanks to all the gophers that participated in OSCON. After the successes of this year we look forward to more Go fun at OSCON 2015.

By Francesc Campoy

Go Concurrency Patterns: Context

29 July 2014

Introduction

In Go servers, each incoming request is handled in its own goroutine. Request handlers often start additional goroutines to access backends such as databases and RPC services. The set of goroutines working on a request typically needs access to request-specific values such as the identity of the end user, authorization tokens, and the request's deadline. When a request is canceled or times out, all the goroutines working on that request should exit quickly so the system can reclaim any resources they are using.

At Google, we developed a context package that makes it easy to pass request-scoped values, cancelation signals, and deadlines across API boundaries to all the goroutines involved in handling a request. The package is publicly available as code.google.com/p/go.net/context. This article describes how to use the package and provides a complete working example.

Context

The core of the context package is the Context type:

// A Context carries a deadline, cancelation signal, and request-scoped values
// across API boundaries. Its methods are safe for simultaneous use by multiple
// goroutines.
type Context interface {
    // Done returns a channel that is closed when this Context is canceled
    // or times out.
    Done() <-chan struct{}

    // Err indicates why this context was canceled, after the Done channel
    // is closed.
    Err() error

    // Deadline returns the time when this Context will be canceled, if any.
    Deadline() (deadline time.Time, ok bool)

    // Value returns the value associated with key or nil if none.
    Value(key interface{}) interface{}
}

(This description is condensed; the godoc is authoritative.)

The Done method returns a channel that acts as a cancelation signal to functions running on behalf of the Context: when the channel is closed, the functions should abandon their work and return. The Err method returns an error indicating why the Context was canceled. The Pipelines and Cancelation article discusses the Done channel idiom in more detail.

A Context does not have a Cancel method for the same reason the Done channel is receive-only: the function receiving a cancelation signal is usually not the one that sends the signal. In particular, when a parent operation starts goroutines for sub-operations, those sub-operations should not be able to cancel the parent. Instead, the WithCancel function (described below) provides a way to cancel a new Context value.

A Context is safe for simultaneous use by multiple goroutines. Code can pass a single Context to any number of goroutines and cancel that Context to signal all of them.

The Deadline method allows functions to determine whether they should start work at all; if too little time is left, it may not be worthwhile. Code may also use a deadline to set timeouts for I/O operations.

Value allows a Context to carry request-scoped data. That data must be safe for simultaneous use by multiple goroutines.

Derived contexts

The context package provides functions to derive new Context values from existing ones. These values form a tree: when a Context is canceled, all Contexts derived from it are also canceled.

Background is the root of any Context tree; it is never canceled:

// Background returns an empty Context. It is never canceled, has no deadline,
// and has no values. Background is typically used in main, init, and tests,
// and as the top-level Context for incoming requests.
func Background() Context

WithCancel and WithTimeout return derived Context values that can be canceled sooner than the parent Context. The Context associated with an incoming request is typically canceled when the request handler returns. WithCancel is also useful for canceling redundant requests when using multiple replicas. WithTimeout is useful for setting a deadline on requests to backend servers:

// WithCancel returns a copy of parent whose Done channel is closed as soon as
// parent.Done is closed or cancel is called.
func WithCancel(parent Context) (ctx Context, cancel CancelFunc)

// A CancelFunc cancels a Context.
type CancelFunc func()

// WithTimeout returns a copy of parent whose Done channel is closed as soon as
// parent.Done is closed, cancel is called, or timeout elapses. The new
// Context's Deadline is the sooner of now+timeout and the parent's deadline, if
// any. If the timer is still running, the cancel function releases its
// resources.
func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc)

WithValue provides a way to associate request-scoped values with a Context:

// WithValue returns a copy of parent whose Value method returns val for key.
func WithValue(parent Context, key interface{}, val interface{}) Context

The best way to see how to use the context package is through a worked example.

Example: Google Web Search

Our example is an HTTP server that handles URLs like /search?q=golang&timeout=1s by forwarding the query "golang" to the Google Web Search API and rendering the results. The timeout parameter tells the server to cancel the request after that duration elapses.

The code is split across three packages:

  • server provides the main function and the handler for /search.
  • userip provides functions for extracting a user IP address from a request and associating it with a Context.
  • google provides the Search function for sending a query to Google.

The server program

The server program handles requests like /search?q=golang by serving the first few Google search results for golang. It registers handleSearch to handle the /search endpoint. The handler creates an initial Context called ctx and arranges for it to be canceled when the handler returns. If the request includes the timeout URL parameter, the Context is canceled automatically when the timeout elapses:

func handleSearch(w http.ResponseWriter, req *http.Request) {
    // ctx is the Context for this handler. Calling cancel closes the
    // ctx.Done channel, which is the cancellation signal for requests
    // started by this handler.
    var (
        ctx    context.Context
        cancel context.CancelFunc
    )
    timeout, err := time.ParseDuration(req.FormValue("timeout"))
    if err == nil {
        // The request has a timeout, so create a context that is
        // canceled automatically when the timeout expires.
        ctx, cancel = context.WithTimeout(context.Background(), timeout)
    } else {
        ctx, cancel = context.WithCancel(context.Background())
    }
    defer cancel() // Cancel ctx as soon as handleSearch returns.

The handler extracts the query from the request and extracts the client's IP address by calling on the userip package. The client's IP address is needed for backend requests, so handleSearch attaches it to ctx:

    // Check the search query.
    query := req.FormValue("q")
    if query == "" {
        http.Error(w, "no query", http.StatusBadRequest)
        return
    }

    // Store the user IP in ctx for use by code in other packages.
    userIP, err := userip.FromRequest(req)
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    ctx = userip.NewContext(ctx, userIP)

The handler calls google.Search with ctx and the query:

    // Run the Google search and print the results.
    start := time.Now()
    results, err := google.Search(ctx, query)
    elapsed := time.Since(start)

If the search succeeds, the handler renders the results:

    if err := resultsTemplate.Execute(w, struct {
        Results          google.Results
        Timeout, Elapsed time.Duration
    }{
        Results: results,
        Timeout: timeout,
        Elapsed: elapsed,
    }); err != nil {
        log.Print(err)
        return
    }

Package userip

The userip package provides functions for extracting a user IP address from a request and associating it with a Context. A Context provides a key-value mapping, where the keys and values are both of type interface{}. Key types must support equality, and values must be safe for simultaneous use by multiple goroutines. Packages like userip hide the details of this mapping and provide strongly-typed access to a specific Context value.

To avoid key collisions, userip defines an unexported type key and uses a value of this type as the context key:

// The key type is unexported to prevent collisions with context keys defined in
// other packages.
type key int

// userIPkey is the context key for the user IP address.  Its value of zero is
// arbitrary.  If this package defined other context keys, they would have
// different integer values.
const userIPKey key = 0

FromRequest extracts a userIP value from an http.Request:

func FromRequest(req *http.Request) (net.IP, error) {
    ip, _, err := net.SplitHostPort(req.RemoteAddr)
    if err != nil {
        return nil, fmt.Errorf("userip: %q is not IP:port", req.RemoteAddr)
    }

NewContext returns a new Context that carries a provided userIP value:

func NewContext(ctx context.Context, userIP net.IP) context.Context {
    return context.WithValue(ctx, userIPKey, userIP)
}

FromContext extracts a userIP from a Context:

func FromContext(ctx context.Context) (net.IP, bool) {
    // ctx.Value returns nil if ctx has no value for the key;
    // the net.IP type assertion returns ok=false for nil.
    userIP, ok := ctx.Value(userIPKey).(net.IP)
    return userIP, ok
}

Package google

The google.Search function makes an HTTP request to the Google Web Search API and parses the JSON-encoded result. It accepts a Context parameter ctx and returns immediately if ctx.Done is closed while the request is in flight.

The Google Web Search API request includes the search query and the user IP as query parameters:

func Search(ctx context.Context, query string) (Results, error) {
    // Prepare the Google Search API request.
    req, err := http.NewRequest("GET", "https://ajax.googleapis.com/ajax/services/search/web?v=1.0", nil)
    if err != nil {
        return nil, err
    }
    q := req.URL.Query()
    q.Set("q", query)

    // If ctx is carrying the user IP address, forward it to the server.
    // Google APIs use the user IP to distinguish server-initiated requests
    // from end-user requests.
    if userIP, ok := userip.FromContext(ctx); ok {
        q.Set("userip", userIP.String())
    }
    req.URL.RawQuery = q.Encode()

Search uses a helper function, httpDo, to issue the HTTP request and cancel it if ctx.Done is closed while the request or response is being processed. Search passes a closure to httpDo handle the HTTP response:

    var results Results
    err = httpDo(ctx, req, func(resp *http.Response, err error) error {
        if err != nil {
            return err
        }
        defer resp.Body.Close()

        // Parse the JSON search result.
        // https://developers.google.com/web-search/docs/#fonje
        var data struct {
            ResponseData struct {
                Results []struct {
                    TitleNoFormatting string
                    URL               string
                }
            }
        }
        if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
            return err
        }
        for _, res := range data.ResponseData.Results {
            results = append(results, Result{Title: res.TitleNoFormatting, URL: res.URL})
        }
        return nil
    })
    // httpDo waits for the closure we provided to return, so it's safe to
    // read results here.
    return results, err

The httpDo function runs the HTTP request and processes its response in a new goroutine. It cancels the request if ctx.Done is closed before the goroutine exits:

func httpDo(ctx context.Context, req *http.Request, f func(*http.Response, error) error) error {
    // Run the HTTP request in a goroutine and pass the response to f.
    tr := &http.Transport{}
    client := &http.Client{Transport: tr}
    c := make(chan error, 1)
    go func() { c <- f(client.Do(req)) }()
    select {
    case <-ctx.Done():
        tr.CancelRequest(req)
        <-c // Wait for f to return.
        return ctx.Err()
    case err := <-c:
        return err
    }
}

Adapting code for Contexts

Many server frameworks provide packages and types for carrying request-scoped values. We can define new implementations of the Context interface to bridge between code using existing frameworks and code that expects a Context parameter.

For example, Gorilla's github.com/gorilla/context package allows handlers to associate data with incoming requests by providing a mapping from HTTP requests to key-value pairs. In gorilla.go, we provide a Context implementation whose Value method returns the values associated with a specific HTTP request in the Gorilla package.

Other packages have provided cancelation support similar to Context. For example, Tomb provides a Kill method that signals cancelation by closing a Dying channel. Tomb also provides methods to wait for those goroutines to exit, similar to sync.WaitGroup. In tomb.go, we provide a Context implementation that is canceled when either its parent Context is canceled or a provided Tomb is killed.

Conclusion

At Google, we require that Go programmers pass a Context parameter as the first argument to every function on the call path between incoming and outgoing requests. This allows Go code developed by many different teams to interoperate well. It provides simple control over timeouts and cancelation and ensures that critical values like security credentials transit Go programs properly.

Server frameworks that want to build on Context should provide implementations of Context to bridge between their packages and those that expect a Context parameter. Their client libraries would then accept a Context from the calling code. By establishing a common interface for request-scoped data and cancelation, Context makes it easier for package developers to share code for creating scalable services.

By Sameer Ajmani

Go will be at OSCON 2014

15 July 2014

OSCON, the Open Source Convention, is taking place from July 20th to the 29th in Portland, Oregon and Go will be central to many talks. If you are attending make sure you add these to your personal schedule.

On Monday you'll have the chance to learn Go in these two tutorials:

During the rest of the week you can hear how different projects use Go:

And if you have any questions come to the Go office hours on Wednesday or come anytime by the Google booth.

See you at OSCON!

By Francesc Campoy

See the index for more articles.