The Go Blog

Errors are values

12 January 2015

A common point of discussion among Go programmers, especially those new to the language, is how to handle errors. The conversation often turns into a lament at the number of times the sequence

if err != nil {
    return err
}

shows up. We recently scanned all the open source projects we could find and discovered that this snippet occurs only once per page or two, less often than some would have you believe. Still, if the perception persists that one must type

if err != nil

all the time, something must be wrong, and the obvious target is Go itself.

This is unfortunate, misleading, and easily corrected. Perhaps what is happening is that programmers new to Go ask, "How does one handle errors?", learn this pattern, and stop there. In other languages, one might use a try-catch block or other such mechanism to handle errors. Therefore, the programmer thinks, when I would have used a try-catch in my old language, I will just type if err != nil in Go. Over time the Go code collects many such snippets, and the result feels clumsy.

Regardless of whether this explanation fits, it is clear that these Go programmers miss a fundamental point about errors: Errors are values.

Values can be programmed, and since errors are values, errors can be programmed.

Of course a common statement involving an error value is to test whether it is nil, but there are countless other things one can do with an error value, and application of some of those other things can make your program better, eliminating much of the boilerplate that arises if every error is checked with a rote if statement.

Here's a simple example from the bufio package's Scanner type. Its Scan method performs the underlying I/O, which can of course lead to an error. Yet the Scan method does not expose an error at all. Instead, it returns a boolean, and a separate method, to be run at the end of the scan, reports whether an error occurred. Client code looks like this:

scanner := bufio.NewScanner(input)
for scanner.Scan() {
    token := scanner.Text()
    // process token
}
if err := scanner.Err(); err != nil {
    // process the error
}

Sure, there is a nil check for an error, but it appears and executes only once. The Scan method could instead have been defined as

func (s *Scanner) Scan() (token []byte, error)

and then the example user code might be (depending on how the token is retrieved),

scanner := bufio.NewScanner(input)
for {
    token, err := scanner.Scan()
    if err != nil {
        return err // or maybe break
    }
    // process token
}

This isn't very different, but there is one important distinction. In this code, the client must check for an error on every iteration, but in the real Scanner API, the error handling is abstracted away from the key API element, which is iterating over tokens. With the real API, the client's code therefore feels more natural: loop until done, then worry about errors. Error handling does not obscure the flow of control.

Under the covers what's happening, of course, is that as soon as Scan encounters an I/O error, it records it and returns false. A separate method, Err, reports the error value when the client asks. Trivial though this is, it's not the same as putting

if err != nil

everywhere or asking the client to check for an error after every token. It's programming with error values. Simple programming, yes, but programming nonetheless.

It's worth stressing that whatever the design, it's critical that the program check the errors however they are exposed. The discussion here is not about how to avoid checking errors, it's about using the language to handle errors with grace.

The topic of repetitive error-checking code arose when I attended the autumn 2014 GoCon in Tokyo. An enthusiastic gopher, who goes by @jxck_ on Twitter, echoed the familiar lament about error checking. He had some code that looked schematically like this:

_, err = fd.Write(p0[a:b])
if err != nil {
    return err
}
_, err = fd.Write(p1[c:d])
if err != nil {
    return err
}
_, err = fd.Write(p2[e:f])
if err != nil {
    return err
}
// and so on

It is very repetitive. In the real code, which was longer, there is more going on so it's not easy to just refactor this using a helper function, but in this idealized form, a function literal closing over the error variable would help:

var err error
write := func(buf []byte) {
    if err != nil {
        return
    }
    _, err = w.Write(buf)
}
write(p0[a:b])
write(p1[c:d])
write(p2[e:f])
// and so on
if err != nil {
    return err
}

This pattern works well, but requires a closure in each function doing the writes; a separate helper function is clumsier to use because the err variable needs to be maintained across calls (try it).

We can make this cleaner, more general, and reusable by borrowing the idea from the Scan method above. I mentioned this technique in our discussion but @jxck_ didn't see how to apply it. After a long exchange, hampered somewhat by a language barrier, I asked if I could just borrow his laptop and show him by typing some code.

I defined an object called an errWriter, something like this:

type errWriter struct {
    w   io.Writer
    err error
}

and gave it one method, write. It doesn't need to have the standard Write signature, and it's lower-cased in part to highlight the distinction. The write method calls the Write method of the underlying Writer and records the first error for future reference:

func (ew *errWriter) write(buf []byte) {
    if ew.err != nil {
        return
    }
    _, ew.err = ew.w.Write(buf)
}

As soon as an error occurs, the write method becomes a no-op but the error value is saved.

Given the errWriter type and its write method, the code above can be refactored:

ew := &errWriter{w: fd}
ew.write(p0[a:b])
ew.write(p1[c:d])
ew.write(p2[e:f])
// and so on
if ew.err != nil {
    return ew.err
}

This is cleaner, even compared to the use of a closure, and also makes the actual sequence of writes being done easier to see on the page. There is no clutter any more. Programming with error values (and interfaces) has made the code nicer.

It's likely that some other piece of code in the same package can build on this idea, or even use errWriter directly.

Also, once errWriter exists, there's more it could do to help, especially in less artificial examples. It could accumulate the byte count. It could coalesce writes into a single buffer that can then be transmitted atomically. And much more.

In fact, this pattern appears often in the standard library. The archive/zip and net/http packages use it. More salient to this discussion, the bufio package's Writer is actually an implementation of the errWriter idea. Although bufio.Writer.Write returns an error, that is mostly about honoring the io.Writer interface. The Write method of bufio.Writer behaves just like our errWriter.write method above, with Flush reporting the error, so our example could be written like this:

b := &bufio.NewWriter(fd)
b.Write(p0[a:b])
b.Write(p1[c:d])
b.Write(p2[e:f])
// and so on
if b.Flush() != nil {
    return b.Flush()
}

There is one significant drawback to this approach, at least for some applications: there is no way to know how much of the processing completed before the error occurred. If that information is important, a more fine-grained approach is necessary. Often, though, an all-or-nothing check at the end is sufficient.

We've looked at just one technique for avoiding repetitive error handling code. Keep in mind that the use of errWriter or bufio.Writer isn't the only way to simplify error handling, and this approach is not suitable for all situations. The key lesson, however, is that errors are values and the full power of the Go programming language is available for processing them.

Use the language to simplify your error handling.

But remember: Whatever you do, always check your errors!

Finally, for the full story of my interaction with @jxck_, including a little video he recorded, visit his blog.

By Rob Pike

GothamGo: gophers in the big apple

9 January 2015

Last November more than two hundred gophers from all across the United States got together for the first full-day Go conference in New York City.

The diverse speaker lineup included university students, industry experts, and Go team members.

And good news, everybody! All the talks were recorded and are available:

Two more talks come from the Go meetup in New York City, which met the day before GothamGo:

  • Benchmarking Go by Brian Bulkowski - the founder of Aerospike talks about profiling tools for Go and Linux, and micro benchmarks for goroutines, channels, buffers, and and other Go features.
  • Go Static Analysis Tools by Alan Donovan - a member of the Go team at Google NY gives a guided tour of several static analysis tools designed to help Go programmers understand, navigate , and refactor their code.

Make sure to have a look at all of those in preparation for the FOSDEM Go devroom FOSDEM in Brussels (Belgium) and gophercon.in in Bengaluru (India).

By Francesc Campoy

The Gopher Gala is the first worldwide Go hackathon

7 January 2015

The Gopher Gala is the first Go hackathon at a global scale and will take place from January 23rd through the 25th. The event is organized by the community, supported by the Go team, and sponsored by Google among others.

Fancy Gopher, by Renée French

You can read about the rules of the hackathon here, but if you know about Rails Rumble or Node Knockout you already have a pretty good idea of what to expect.

During this event gophers from all around the globe will form teams to build great applications using Go as the main tool. Afterwards, all the participants will vote for their favorite applications and the 20 highest voted applications will be ranked by a jury of renowned names from the community, including some members of the Go core team. More information on the judging phase can be found here.

And in case you needed one more reason to get involved there will be prizes!

"What prizes?" you ask. Well, that’s a secret until January 10th but we’re pretty sure you won’t be disappointed.

By Francesc Campoy

Generating code

22 December 2014

A property of universal computation—Turing completeness—is that a computer program can write a computer program. This is a powerful idea that is not appreciated as often as it might be, even though it happens frequently. It's a big part of the definition of a compiler, for instance. It's also how the go test command works: it scans the packages to be tested, writes out a Go program containing a test harness customized for the package, and then compiles and runs it. Modern computers are so fast this expensive-sounding sequence can complete in a fraction of a second.

There are lots of other examples of programs that write programs. Yacc, for instance, reads in a description of a grammar and writes out a program to parse that grammar. The protocol buffer "compiler" reads an interface description and emits structure definitions, methods, and other support code. Configuration tools of all sorts work like this too, examining metadata or the environment and emitting scaffolding customized to the local state.

Programs that write programs are therefore important elements in software engineering, but programs like Yacc that produce source code need to be integrated into the build process so their output can be compiled. When an external build tool like Make is being used, this is usually easy to do. But in Go, whose go tool gets all necessary build information from the Go source, there is a problem. There is simply no mechanism to run Yacc from the go tool alone.

Until now, that is.

The latest Go release, 1.4, includes a new command that makes it easier to run such tools. It's called go generate, and it works by scanning for special comments in Go source code that identify general commands to run. It's important to understand that go generate is not part of go build. It contains no dependency analysis and must be run explicitly before running go build. It is intended to be used by the author of the Go package, not its clients.

The go generate command is easy to use. As a warmup, here's how to use it to generate a Yacc grammar. Say you have a Yacc input file called gopher.y that defines a grammar for your new language. To produce the Go source file implementing the grammar, you would normally invoke the standard Go version of Yacc like this:

go tool yacc -o gopher.go -p parser gopher.y

The -o option names the output file while -p specifies the package name.

To have go generate drive the process, in any one of the regular (non-generated) .go files in the same directory, add this comment anywhere in the file:

//go:generate go tool yacc -o gopher.go -p parser gopher.y

This text is just the command above prefixed by a special comment recognized by go generate. The comment must start at the beginning of the line and have no spaces between the // and the go:generate. After that marker, the rest of the line specifies a command for go generate to run.

Now run it. Change to the source directory and run go generate, then go build and so on:

$ cd $GOPATH/myrepo/gopher
$ go generate
$ go build
$ go test

That's it. Assuming there are no errors, the go generate command will invoke yacc to create gopher.go, at which point the directory holds the full set of Go source files, so we can build, test, and work normally. Every time gopher.y is modified, just rerun go generate to regenerate the parser.

For more details about how go generate works, including options, environment variables, and so on, see the design document.

Go generate does nothing that couldn't be done with Make or some other build mechanism, but it comes with the go tool—no extra installation required—and fits nicely into the Go ecosystem. Just keep in mind that it is for package authors, not clients, if only for the reason that the program it invokes might not be available on the target machine. Also, if the containing package is intended for import by go get, once the file is generated (and tested!) it must be checked into the source code repository to be available to clients.

Now that we have it, let's use it for something new. As a very different example of how go generate can help, there is a new program available in the golang.org/x/tools repository called stringer. It automatically writes string methods for sets of integer constants. It's not part of the released distribution, but it's easy to install:

$ go get golang.org/x/tools/cmd/stringer

Here's an example from the documentation for stringer. Imagine we have some code that contains a set of integer constants defining different types of pills:

package painkiller

type Pill int

const (
    Placebo Pill = iota
    Aspirin
    Ibuprofen
    Paracetamol
    Acetaminophen = Paracetamol
)

For debugging, we'd like these constants to pretty-print themselves, which means we want a method with signature,

func (p Pill) String() string

It's easy to write one by hand, perhaps like this:

func (p Pill) String() string {
    switch p {
    case Placebo:
        return "Placebo"
    case Aspirin:
        return "Aspirin"
    case Ibuprofen:
        return "Ibuprofen"
    case Paracetamol: // == Acetaminophen
        return "Paracetamol"
    }
    return fmt.Sprintf("Pill(%d)", p)
}

There are other ways to write this function, of course. We could use a slice of strings indexed by Pill, or a map, or some other technique. Whatever we do, we need to maintain it if we change the set of pills, and we need to make sure it's correct. (The two names for paracetamol make this trickier than it might otherwise be.) Plus the very question of which approach to take depends on the types and values: signed or unsigned, dense or sparse, zero-based or not, and so on.

The stringer program takes care of all these details. Although it can be run in isolation, it is intended to be driven by go generate. To use it, add a generate comment to the source, perhaps near the type definition:

//go:generate stringer -type=Pill

This rule specifies that go generate should run the stringer tool to generate a String method for type Pill. The output is automatically written to pill_string.go (a default we could override with the -output flag).

Let's run it:

$ go generate
$ cat pill_string.go
// generated by stringer -type Pill pill.go; DO NOT EDIT

package pill

import "fmt"

const _Pill_name = "PlaceboAspirinIbuprofenParacetamol"

var _Pill_index = [...]uint8{0, 7, 14, 23, 34}

func (i Pill) String() string {
    if i < 0 || i+1 >= Pill(len(_Pill_index)) {
        return fmt.Sprintf("Pill(%d)", i)
    }
    return _Pill_name[_Pill_index[i]:_Pill_index[i+1]]
}
$

Every time we change the definition of Pill or the constants, all we need to do is run

$ go generate

to update the String method. And of course if we've got multiple types set up this way in the same package, that single command will update all their String methods with a single command.

There's no question the generated method is ugly. That's OK, though, because humans don't need to work on it; machine-generated code is often ugly. It's working hard to be efficient. All the names are smashed together into a single string, which saves memory (only one string header for all the names, even if there are zillions of them). Then an array, _Pill_index, maps from value to name by a simple, efficient technique. Note too that _Pill_index is an array (not a slice; one more header eliminated) of uint8, the smallest integer sufficient to span the space of values. If there were more values, or there were negatives ones, the generated type of _Pill_index might change to uint16 or int8: whatever works best.

The approach used by the methods printed by stringer varies according to the properties of the constant set. For instance, if the constants are sparse, it might use a map. Here's a trivial example based on a constant set representing powers of two:

const _Power_name = "p0p1p2p3p4p5..."

var _Power_map = map[Power]string{
    1:    _Power_name[0:2],
    2:    _Power_name[2:4],
    4:    _Power_name[4:6],
    8:    _Power_name[6:8],
    16:   _Power_name[8:10],
    32:   _Power_name[10:12],
    ...,
}

func (i Power) String() string {
    if str, ok := _Power_map[i]; ok {
        return str
    }
    return fmt.Sprintf("Power(%d)", i)
}

In short, generating the method automatically allows us to do a better job than we would expect a human to do.

There are lots of other uses of go generate already installed in the Go tree. Examples include generating Unicode tables in the unicode package, creating efficient methods for encoding and decoding arrays in encoding/gob, producing time zone data in the time package, and so on.

Please use go generate creatively. It's there to encourage experimentation.

And even if you don't, use the new stringer tool to write your String methods for your integer constants. Let the machine do the work.

By Rob Pike

Go 1.4 is released

10 December 2014

Today we announce Go 1.4, the fifth major stable release of Go, arriving six months after our previous major release Go 1.3. It contains a small language change, support for more operating systems and processor architectures, and improvements to the tool chain and libraries. As always, Go 1.4 keeps the promise of compatibility, and almost everything will continue to compile and run without change when moved to 1.4. For the full details, see the Go 1.4 release notes.

The most notable new feature in this release is official support for Android. Using the support in the core and the libraries in the golang.org/x/mobile repository, it is now possible to write simple Android apps using only Go code. At this stage, the support libraries are still nascent and under heavy development. Early adopters should expect a bumpy ride, but we welcome the community to get involved.

The language change is a tweak to the syntax of for-range loops. You may now write "for range s {" to loop over each item from s, without having to assign the value, loop index, or map key. See the release notes for details.

The go command has a new subcommand, go generate, to automate the running of tools to generate source code before compilation. For example, it can be used to automate the generation of String methods for typed constants using the new stringer tool. For more information, see the design document.

Most programs will run about the same speed or slightly faster in 1.4 than in 1.3; some will be slightly slower. There are many changes, making it hard to be precise about what to expect. See the release notes for more discussion.

And, of course, there are many more improvements and bug fixes.

In case you missed it, a few weeks ago the sub-repositories were moved to new locations. For example, the go.tools packages are now imported from "golang.org/x/tools". See the announcement post for details.

This release also coincides with the project's move from Mercurial to Git (for source control), Rietveld to Gerrit (for code review), and Google Code to Github (for issue tracking and wiki). The move affects the core Go repository and its sub-repositories. You can find the canonical Git repositories at go.googlesource.com, and the issue tracker and wiki at the golang/go GitHub repo.

While development has already moved over to the new infrastructure, for the 1.4 release we still recommend that users who install from source use the Mercurial repositories.

For App Engine users, Go 1.4 is now available for beta testing. See the announcement for details.

From all of us on the Go team, please enjoy Go 1.4, and have a happy holiday season.

By Andrew Gerrand

See the index for more articles.