The Go Blog

Testable Examples in Go

7 May 2015

Introduction

Godoc examples are snippets of Go code that are displayed as package documentation and that are verified by running them as tests. They can also be run by a user visiting the godoc web page for the package and clicking the associated "Run" button.

Having executable documentation for a package guarantees that the information will not go out of date as the API changes.

The standard library includes many such examples (see the strings package, for instance).

This article explains how to write your own example functions.

Examples are tests

Examples are compiled (and optionally executed) as part of a package's test suite.

As with typical tests, examples are functions that reside in a package's _test.go files. Unlike normal test functions, though, example functions take no arguments and begin with the word Example instead of Test.

The stringutil package is part of the Go example repository. Here's an example that demonstrates its Reverse function:

package stringutil_test

import (
    "fmt"

    "github.com/golang/example/stringutil"
)

func ExampleReverse() {
    fmt.Println(stringutil.Reverse("hello"))
    // Output: olleh
}

This code might live in example_test.go in the stringutil directory.

Godoc will present this example alongside the Reverse function's documentation:

Running the package's test suite, we can see the example function is executed with no further arrangement from us:

$ go test -v
=== RUN TestReverse
--- PASS: TestReverse (0.00s)
=== RUN: ExampleReverse
--- PASS: ExampleReverse (0.00s)
PASS
ok      github.com/golang/example/stringutil    0.009s

Output comments

What does it mean that the ExampleReverse function "passes"?

As it executes the example, the testing framework captures data written to standard output and then compares the output against the example's "Output:" comment. The test passes if the test's output matches its output comment.

To see a failing example we can change the output comment text to something obviously incorrect

func ExampleReverse() {
    fmt.Println(stringutil.Reverse("hello"))
    // Output: golly
}

and run the tests again:

$ go test
--- FAIL: ExampleReverse (0.00s)
got:
olleh
want:
golly
FAIL

If we remove the output comment entirely

func ExampleReverse() {
    fmt.Println(stringutil.Reverse("hello"))
}

then the example function is compiled but not executed:

$ go test -v
=== RUN TestReverse
--- PASS: TestReverse (0.00s)
PASS
ok      github.com/golang/example/stringutil    0.009s

Examples without output comments are useful for demonstrating code that cannot run as unit tests, such as that which accesses the network, while guaranteeing the example at least compiles.

Example function names

Godoc uses a naming convention to associate an example function with a package-level identifier.

func ExampleFoo()     // documents the Foo function or type
func ExampleBar_Qux() // documents the Qux method of type Bar
func Example()        // documents the package as a whole

Following this convention, godoc displays the ExampleReverse example alongside the documentation for the Reverse function.

Multiple examples can be provided for a given identifier by using a suffix beginning with an underscore followed by a lowercase letter. Each of these examples documents the Reverse function:

func ExampleReverse()
func ExampleReverse_second()
func ExampleReverse_third()

Larger examples

Sometimes we need more than just a function to write a good example.

For instance, to demonstrate the sort package we should show an implementation of sort.Interface. Since methods cannot be declared inside a function body, the example must include some context in addition to the example function.

To achieve this we can use a "whole file example." A whole file example is a file that ends in _test.go and contains exactly one example function, no test or benchmark functions, and at least one other package-level declaration. When displaying such examples godoc will show the entire file.

Here is a whole file example from the sort package:

package sort_test

import (
    "fmt"
    "sort"
)

type Person struct {
    Name string
    Age  int
}

func (p Person) String() string {
    return fmt.Sprintf("%s: %d", p.Name, p.Age)
}

// ByAge implements sort.Interface for []Person based on
// the Age field.
type ByAge []Person

func (a ByAge) Len() int           { return len(a) }
func (a ByAge) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
func (a ByAge) Less(i, j int) bool { return a[i].Age < a[j].Age }

func Example() {
    people := []Person{
        {"Bob", 31},
        {"John", 42},
        {"Michael", 17},
        {"Jenny", 26},
    }

    fmt.Println(people)
    sort.Sort(ByAge(people))
    fmt.Println(people)

    // Output:
    // [Bob: 31 John: 42 Michael: 17 Jenny: 26]
    // [Michael: 17 Jenny: 26 Bob: 31 John: 42]
}

A package can contain multiple whole file examples; one example per file. Take a look at the sort package's source code to see this in practice.

Conclusion

Godoc examples are a great way to write and maintain code as documentation. They also present editable, working, runnable examples your users can build on. Use them!

By Andrew Gerrand

Package names

4 February 2015

Introduction

Go code is organized into packages. Within a package, code can refer to any identifier (name) defined within, while clients of the package may only reference the package's exported types, functions, constants, and variables. Such references always include the package name as a prefix: foo.Bar refers to the exported name Bar in the imported package named foo.

Good package names make code better. A package's name provides context for its contents, making it easier for clients to understand what the package is for and how to use it. The name also helps package maintainers determine what does and does not belong in the package as it evolves. Well-named packages make it easier to find the code you need.

Effective Go provides guidelines for naming packages, types, functions, and variables. This article expands on that discussion and surveys names found in the standard library. It also discusses bad package names and how to fix them.

Package names

Good package names are short and clear. They are lower case, with no under_scores or mixedCaps. They are often simple nouns, such as:

  • time (provides functionality for measuring and displaying time)
  • list (implements a doubly linked list)
  • http (provides HTTP client and server implementations)

The style of names typical of another language might not be idiomatic in a Go program. Here are two examples of names that might be good style in other languages but do not fit well in Go:

  • computeServiceClient
  • priority_queue

A Go package may export several types and functions. For example, a compute package could export a Client type with methods for using the service as well as functions for partitioning a compute task across several clients.

Abbreviate judiciously. Package names may be abbreviated when the abbreviation is familiar to the programmer. Widely-used packages often have compressed names:

  • strconv (string conversion)
  • syscall (system call)
  • fmt (formatted I/O)

On the other hand, if abbreviating a package name makes it ambiguous or unclear, don't do it.

Don't steal good names from the user. Avoid giving a package a name that is commonly used in client code. For example, the buffered I/O package is called bufio, not buf, since buf is a good variable name for a buffer.

Naming package contents

A package name and its contents' names are coupled, since client code uses them together. When designing a package, take the client's point of view.

Avoid stutter. Since client code uses the package name as a prefix when referring to the package contents, the names for those contents need not repeat the package name. The HTTP server provided by the http package is called Server, not HTTPServer. Client code refers to this type as http.Server, so there is no ambiguity.

Simplify function names. When a function in package pkg returns a value of type pkg.Pkg (or *pkg.Pkg), the function name can often omit the type name without confusion:

start := time.Now()                                  // start is a time.Time
t, err := time.Parse(time.Kitchen, "6:06PM")         // t is a time.Time
ctx = context.WithTimeout(ctx, 10*time.Millisecond)  // ctx is a context.Context
ip, ok := userip.FromContext(ctx)                    // ip is a net.IP

A function named New in package pkg returns a value of type pkg.Pkg. This is a standard entry point for client code using that type:

q := list.New()  // q is a *list.List

When a function returns a value of type pkg.T, where T is not Pkg, the function name may include T to make client code easier to understand. A common situation is a package with multiple New-like functions:

d, err := time.ParseDuration("10s")  // d is a time.Duration
elapsed := time.Since(start)         // elapsed is a time.Duration
ticker := time.NewTicker(d)          // ticker is a *time.Ticker
timer := time.NewTimer(d)            // timer is a *time.Timer

Types in different packages can have the same name, because from the client's point of view such names are discriminated by the package name. For example, the standard library includes several types named Reader, including jpeg.Reader, bufio.Reader, and csv.Reader. Each package name fits with Reader to yield a good type name.

If you cannot come up with a package name that's a meaningful prefix for the package's contents, the package abstraction boundary may be wrong. Write code that uses your package as a client would, and restructure your packages if the result seems poor. This approach will yield packages that are easier for clients to understand and for the package developers to maintain.

Package paths

A Go package has both a name and a path. The package name is specified in the package statement of its source files; client code uses it as the prefix for the package's exported names. Client code uses the package path when importing the package. By convention, the last element of the package path is the package name:

import (
    "fmt"                       // package fmt
    "os/exec"                   // package exec
    "golang.org/x/net/context"  // package context
)

Build tools map package paths onto directories. The go tool uses the GOPATH environment variable to find the source files for path "github.com/user/hello" in directory $GOPATH/src/github.com/user/hello. (This situation should be familiar, of course, but it's important to be clear about the terminology and structure of packages.)

Directories. The standard library uses like directories crypto, container, encoding, and image to group packages for related protocols and algorithms. There is no actual relationship among the packages in one of these directories; a directory just provides a way to arrange the files. Any package can import any other package provided the import does not create a cycle.

Just as types in different packages can have the same name without ambiguity, packages in different directories can have the same name. For example, runtime/pprof provides profiling data in the format expected by the pprof profiling tool, while net/http/pprof provides HTTP endpoints to present profiling data in this format. Client code uses the package path to import the package, so there is no confusion. If a source file needs to import both pprof packages, it can rename one or both locally. When renaming an imported package, the local name should follow the same guidelines as package names (lower case, no under_scores or mixedCaps).

Bad package names

Bad package names make code harder to navigate and maintain. Here are some guidelines for recognizing and fixing bad names.

Avoid meaningless package names. Packages named util, common, or misc provide clients with no sense of what the package contains. This makes it harder for clients to use the package and makes it harder for maintainers to keep the package focused. Over time, they accumulate dependencies that can make compilation significantly and unnecessarily slower, especially in large programs. And since such package names are generic, they are more likely to collide with other packages imported by client code, forcing clients to invent names to distinguish them.

Break up generic packages. To fix such packages, look for types and functions with common name elements and pull them into their own package. For example, if you have

package util
func NewStringSet(...string) map[string]bool {...}
func SortStringSet(map[string]bool) []string {...}

then client code looks like

set := util.NewStringSet("c", "a", "b")
fmt.Println(util.SortStringSet(set))

Pull these functions out of util into a new package, choosing a name that fits the contents:

package stringset
func New(...string) map[string]bool {...}
func Sort(map[string]bool) []string {...}

then the client code becomes

set := stringset.New("c", "a", "b")
fmt.Println(stringset.Sort(set))

Once you've made this change, its easier to see how to improve the new package:

package stringset
type Set map[string]bool
func New(...string) Set {...}
func (s Set) Sort() []string {...}

which yields even simpler client code:

set := stringset.New("c", "a", "b")
fmt.Println(set.Sort())

The name of the package is a critical piece of its design. Work to eliminate meaningless package names from your projects.

Don't use a single package for all your APIs. Many well-intentioned programmers put all the interfaces exposed by their program into a single package named api, types, or interfaces, thinking it makes it easier to find the entry points to their code base. This is a mistake. Such packages suffer from the same problems as those named util or common, growing without bound, providing no guidance to users, accumulating dependencies, and colliding with other imports. Break them up, perhaps using directories to separate public packages from implementation.

Avoid unnecessary package name collisions. While packages in different directories may have the same name, packages that are frequently used together should have distinct names. This reduces confusion and the need for local renaming in client code. For the same reason, avoid using the same name as popular standard packages like io or http.

Conclusion

Package names are central to good naming in Go programs. Take the time to choose good package names and organize your code well. This helps clients understand and use your packages and helps maintainers to grow them gracefully.

Further reading

By Sameer Ajmani

Errors are values

12 January 2015

A common point of discussion among Go programmers, especially those new to the language, is how to handle errors. The conversation often turns into a lament at the number of times the sequence

if err != nil {
    return err
}

shows up. We recently scanned all the open source projects we could find and discovered that this snippet occurs only once per page or two, less often than some would have you believe. Still, if the perception persists that one must type

if err != nil

all the time, something must be wrong, and the obvious target is Go itself.

This is unfortunate, misleading, and easily corrected. Perhaps what is happening is that programmers new to Go ask, "How does one handle errors?", learn this pattern, and stop there. In other languages, one might use a try-catch block or other such mechanism to handle errors. Therefore, the programmer thinks, when I would have used a try-catch in my old language, I will just type if err != nil in Go. Over time the Go code collects many such snippets, and the result feels clumsy.

Regardless of whether this explanation fits, it is clear that these Go programmers miss a fundamental point about errors: Errors are values.

Values can be programmed, and since errors are values, errors can be programmed.

Of course a common statement involving an error value is to test whether it is nil, but there are countless other things one can do with an error value, and application of some of those other things can make your program better, eliminating much of the boilerplate that arises if every error is checked with a rote if statement.

Here's a simple example from the bufio package's Scanner type. Its Scan method performs the underlying I/O, which can of course lead to an error. Yet the Scan method does not expose an error at all. Instead, it returns a boolean, and a separate method, to be run at the end of the scan, reports whether an error occurred. Client code looks like this:

scanner := bufio.NewScanner(input)
for scanner.Scan() {
    token := scanner.Text()
    // process token
}
if err := scanner.Err(); err != nil {
    // process the error
}

Sure, there is a nil check for an error, but it appears and executes only once. The Scan method could instead have been defined as

func (s *Scanner) Scan() (token []byte, error)

and then the example user code might be (depending on how the token is retrieved),

scanner := bufio.NewScanner(input)
for {
    token, err := scanner.Scan()
    if err != nil {
        return err // or maybe break
    }
    // process token
}

This isn't very different, but there is one important distinction. In this code, the client must check for an error on every iteration, but in the real Scanner API, the error handling is abstracted away from the key API element, which is iterating over tokens. With the real API, the client's code therefore feels more natural: loop until done, then worry about errors. Error handling does not obscure the flow of control.

Under the covers what's happening, of course, is that as soon as Scan encounters an I/O error, it records it and returns false. A separate method, Err, reports the error value when the client asks. Trivial though this is, it's not the same as putting

if err != nil

everywhere or asking the client to check for an error after every token. It's programming with error values. Simple programming, yes, but programming nonetheless.

It's worth stressing that whatever the design, it's critical that the program check the errors however they are exposed. The discussion here is not about how to avoid checking errors, it's about using the language to handle errors with grace.

The topic of repetitive error-checking code arose when I attended the autumn 2014 GoCon in Tokyo. An enthusiastic gopher, who goes by @jxck_ on Twitter, echoed the familiar lament about error checking. He had some code that looked schematically like this:

_, err = fd.Write(p0[a:b])
if err != nil {
    return err
}
_, err = fd.Write(p1[c:d])
if err != nil {
    return err
}
_, err = fd.Write(p2[e:f])
if err != nil {
    return err
}
// and so on

It is very repetitive. In the real code, which was longer, there is more going on so it's not easy to just refactor this using a helper function, but in this idealized form, a function literal closing over the error variable would help:

var err error
write := func(buf []byte) {
    if err != nil {
        return
    }
    _, err = w.Write(buf)
}
write(p0[a:b])
write(p1[c:d])
write(p2[e:f])
// and so on
if err != nil {
    return err
}

This pattern works well, but requires a closure in each function doing the writes; a separate helper function is clumsier to use because the err variable needs to be maintained across calls (try it).

We can make this cleaner, more general, and reusable by borrowing the idea from the Scan method above. I mentioned this technique in our discussion but @jxck_ didn't see how to apply it. After a long exchange, hampered somewhat by a language barrier, I asked if I could just borrow his laptop and show him by typing some code.

I defined an object called an errWriter, something like this:

type errWriter struct {
    w   io.Writer
    err error
}

and gave it one method, write. It doesn't need to have the standard Write signature, and it's lower-cased in part to highlight the distinction. The write method calls the Write method of the underlying Writer and records the first error for future reference:

func (ew *errWriter) write(buf []byte) {
    if ew.err != nil {
        return
    }
    _, ew.err = ew.w.Write(buf)
}

As soon as an error occurs, the write method becomes a no-op but the error value is saved.

Given the errWriter type and its write method, the code above can be refactored:

ew := &errWriter{w: fd}
ew.write(p0[a:b])
ew.write(p1[c:d])
ew.write(p2[e:f])
// and so on
if ew.err != nil {
    return ew.err
}

This is cleaner, even compared to the use of a closure, and also makes the actual sequence of writes being done easier to see on the page. There is no clutter any more. Programming with error values (and interfaces) has made the code nicer.

It's likely that some other piece of code in the same package can build on this idea, or even use errWriter directly.

Also, once errWriter exists, there's more it could do to help, especially in less artificial examples. It could accumulate the byte count. It could coalesce writes into a single buffer that can then be transmitted atomically. And much more.

In fact, this pattern appears often in the standard library. The archive/zip and net/http packages use it. More salient to this discussion, the bufio package's Writer is actually an implementation of the errWriter idea. Although bufio.Writer.Write returns an error, that is mostly about honoring the io.Writer interface. The Write method of bufio.Writer behaves just like our errWriter.write method above, with Flush reporting the error, so our example could be written like this:

b := bufio.NewWriter(fd)
b.Write(p0[a:b])
b.Write(p1[c:d])
b.Write(p2[e:f])
// and so on
if b.Flush() != nil {
    return b.Flush()
}

There is one significant drawback to this approach, at least for some applications: there is no way to know how much of the processing completed before the error occurred. If that information is important, a more fine-grained approach is necessary. Often, though, an all-or-nothing check at the end is sufficient.

We've looked at just one technique for avoiding repetitive error handling code. Keep in mind that the use of errWriter or bufio.Writer isn't the only way to simplify error handling, and this approach is not suitable for all situations. The key lesson, however, is that errors are values and the full power of the Go programming language is available for processing them.

Use the language to simplify your error handling.

But remember: Whatever you do, always check your errors!

Finally, for the full story of my interaction with @jxck_, including a little video he recorded, visit his blog.

By Rob Pike

GothamGo: gophers in the big apple

9 January 2015

Last November more than two hundred gophers from all across the United States got together for the first full-day Go conference in New York City.

The diverse speaker lineup included university students, industry experts, and Go team members.

And good news, everybody! All the talks were recorded and are available:

Two more talks come from the Go meetup in New York City, which met the day before GothamGo:

  • Benchmarking Go by Brian Bulkowski - the founder of Aerospike talks about profiling tools for Go and Linux, and micro benchmarks for goroutines, channels, buffers, and and other Go features.
  • Go Static Analysis Tools by Alan Donovan - a member of the Go team at Google NY gives a guided tour of several static analysis tools designed to help Go programmers understand, navigate , and refactor their code.

Make sure to have a look at all of those in preparation for the FOSDEM Go devroom FOSDEM in Brussels (Belgium) and gophercon.in in Bengaluru (India).

By Francesc Campoy

The Gopher Gala is the first worldwide Go hackathon

7 January 2015

The Gopher Gala is the first Go hackathon at a global scale and will take place from January 23rd through the 25th. The event is organized by the community, supported by the Go team, and sponsored by Google among others.

Fancy Gopher, by Renée French

You can read about the rules of the hackathon here, but if you know about Rails Rumble or Node Knockout you already have a pretty good idea of what to expect.

During this event gophers from all around the globe will form teams to build great applications using Go as the main tool. Afterwards, all the participants will vote for their favorite applications and the 20 highest voted applications will be ranked by a jury of renowned names from the community, including some members of the Go core team. More information on the judging phase can be found here.

And in case you needed one more reason to get involved there will be prizes!

"What prizes?" you ask. Well, that’s a secret until January 10th but we’re pretty sure you won’t be disappointed.

By Francesc Campoy

See the index for more articles.