Godoc: documenting Go code

The Go project takes documentation seriously. Documentation is a huge part of making software accessible and maintainable. Of course it must be well-written and accurate, but it also must be easy to write and to maintain. Ideally, it should be coupled to the code itself so the documentation evolves along with the code. The easier it is for programmers to produce good documentation, the better for everyone.

To that end, we have developed the godoc documentation tool. This article describes godoc's approach to documentation, and explains how you can use our conventions and tools to write good documentation for your own projects.

Godoc parses Go source code - including comments - and produces documentation as HTML or plain text. The end result is documentation tightly coupled with the code it documents. For example, through godoc's web interface you can navigate from a function's documentation to its implementation with one click.

Godoc is conceptually related to Python's Docstring and Java's Javadoc, but its design is simpler. The comments read by godoc are not language constructs (as with Docstring) nor must they have their own machine-readable syntax (as with Javadoc). Godoc comments are just good comments, the sort you would want to read even if godoc didn't exist.

The convention is simple: to document a type, variable, constant, function, or even a package, write a regular comment directly preceding its declaration, with no intervening blank line. Godoc will then present that comment as text alongside the item it documents. For example, this is the documentation for the fmt package's Fprint function:

// Fprint formats using the default formats for its operands and writes to w.
// Spaces are added between operands when neither is a string.
// It returns the number of bytes written and any write error encountered.
func Fprint(w io.Writer, a ...interface{}) (n int, err error) {

Notice this comment is a complete sentence that begins with the name of the element it describes. This important convention allows us to generate documentation in a variety of formats, from plain text to HTML to UNIX man pages, and makes it read better when tools truncate it for brevity, such as when they extract the first line or sentence.

Comments on package declarations should provide general package documentation. These comments can be short, like the sort package's brief description:

// Package sort provides primitives for sorting slices and user-defined
// collections.
package sort

They can also be detailed like the gob package's overview. That package uses another convention for packages that need large amounts of introductory documentation: the package comment is placed in its own file, doc.go, which contains only those comments and a package clause.

When writing package comments of any size, keep in mind that their first sentence will appear in godoc's package list.

Comments that are not adjacent to a top-level declaration are omitted from godoc's output, with one notable exception. Top-level comments that begin with the word "BUG(who)” are recognized as known bugs, and included in the "Bugs” section of the package documentation. The "who” part should be the user name of someone who could provide more information. For example, this is a known issue from the bytes package:

// BUG(r): The rule Title uses for word boundaries does not handle Unicode punctuation properly.

Godoc treats executable commands somewhat differently. Instead of inspecting the command source code, it looks for a Go source file belonging to the special package "documentation”. The comment on the "package documentation” clause is used as the command's documentation. For example, see the godoc documentation and its corresponding doc.go file.

There are a few formatting rules that Godoc uses when converting comments to HTML:

• Subsequent lines of text are considered part of the same paragraph; you must leave a blank line to separate paragraphs.
• Pre-formatted text must be indented relative to the surrounding comment text (see gob's doc.go for an example).
• URLs will be converted to HTML links; no special markup is necessary.

Note that none of these rules requires you to do anything out of the ordinary.

In fact, the best thing about godoc's minimal approach is how easy it is to use. As a result, a lot of Go code, including all of the standard library, already follows the conventions.

Your own code can present good documentation just by having comments as described above. Any Go packages installed inside $GOROOT/src/pkg and any GOPATH work spaces will already be accessible via godoc's command-line and HTTP interfaces, and you can specify additional paths for indexing via the -path flag or just by running "godoc ." in the source directory. See the godoc documentation for more details.

Gobs of data

To transmit a data structure across a network or to store it in a file, it must be encoded and then decoded again. There are many encodings available, of course: JSON, XML, Google's protocol buffers, and more. And now there's another, provided by Go's gob package.

Why define a new encoding? It's a lot of work and redundant at that. Why not just use one of the existing formats? Well, for one thing, we do! Go has packages supporting all the encodings just mentioned (the protocol buffer package is in a separate repository but it's one of the most frequently downloaded). And for many purposes, including communicating with tools and systems written in other languages, they're the right choice.

But for a Go-specific environment, such as communicating between two servers written in Go, there's an opportunity to build something much easier to use and possibly more efficient.

Gobs work with the language in a way that an externally-defined, language-independent encoding cannot. At the same time, there are lessons to be learned from the existing systems.

Goals

The gob package was designed with a number of goals in mind.

First, and most obvious, it had to be very easy to use. First, because Go has reflection, there is no need for a separate interface definition language or "protocol compiler". The data structure itself is all the package should need to figure out how to encode and decode it. On the other hand, this approach means that gobs will never work as well with other languages, but that's OK: gobs are unashamedly Go-centric.

Efficiency is also important. Textual representations, exemplified by XML and JSON, are too slow to put at the center of an efficient communications network. A binary encoding is necessary.

Gob streams must be self-describing. Each gob stream, read from the beginning, contains sufficient information that the entire stream can be parsed by an agent that knows nothing a priori about its contents. This property means that you will always be able to decode a gob stream stored in a file, even long after you've forgotten what data it represents.

There were also some things to learn from our experiences with Google protocol buffers.

Protocol buffer misfeatures

Protocol buffers had a major effect on the design of gobs, but have three features that were deliberately avoided. (Leaving aside the property that protocol buffers aren't self-describing: if you don't know the data definition used to encode a protocol buffer, you might not be able to parse it.)

First, protocol buffers only work on the data type we call a struct in Go. You can't encode an integer or array at the top level, only a struct with fields inside it. That seems a pointless restriction, at least in Go. If all you want to send is an array of integers, why should you have to put it into a struct first?

Next, a protocol buffer definition may specify that fields T.x and T.y are required to be present whenever a value of type T is encoded or decoded. Although such required fields may seem like a good idea, they are costly to implement because the codec must maintain a separate data structure while encoding and decoding, to be able to report when required fields are missing. They're also a maintenance problem. Over time, one may want to modify the data definition to remove a required field, but that may cause existing clients of the data to crash. It's better not to have them in the encoding at all. (Protocol buffers also have optional fields. But if we don't have required fields, all fields are optional and that's that. There will be more to say about optional fields a little later.)

The third protocol buffer misfeature is default values. If a protocol buffer omits the value for a "defaulted" field, then the decoded structure behaves as if the field were set to that value. This idea works nicely when you have getter and setter methods to control access to the field, but is harder to handle cleanly when the container is just a plain idiomatic struct. Required fields are also tricky to implement: where does one define the default values, what types do they have (is text UTF-8? uninterpreted bytes? how many bits in a float?) and despite the apparent simplicity, there were a number of complications in their design and implementation for protocol buffers. We decided to leave them out of gobs and fall back to Go's trivial but effective defaulting rule: unless you set something otherwise, it has the "zero value" for that type - and it doesn't need to be transmitted.

So gobs end up looking like a sort of generalized, simplified protocol buffer. How do they work?

Values

The encoded gob data isn't about int8s and uint16s. Instead, somewhat analogous to constants in Go, its integer values are abstract, sizeless numbers, either signed or unsigned. When you encode an int8, its value is transmitted as an unsized, variable-length integer. When you encode an int64, its value is also transmitted as an unsized, variable-length integer. (Signed and unsigned are treated distinctly, but the same unsized-ness applies to unsigned values too.) If both have the value 7, the bits sent on the wire will be identical. When the receiver decodes that value, it puts it into the receiver's variable, which may be of arbitrary integer type. Thus an encoder may send a 7 that came from an int8, but the receiver may store it in an int64. This is fine: the value is an integer and as a long as it fits, everything works. (If it doesn't fit, an error results.) This decoupling from the size of the variable gives some flexibility to the encoding: we can expand the type of the integer variable as the software evolves, but still be able to decode old data.

This flexibility also applies to pointers. Before transmission, all pointers are flattened. Values of type int8, *int8, **int8, ****int8, etc. are all transmitted as an integer value, which may then be stored in int of any size, or *int, or ******int, etc. Again, this allows for flexibility.

Flexibility also happens because, when decoding a struct, only those fields that are sent by the encoder are stored in the destination. Given the value

type T struct{ X, Y, Z int } // Only exported fields are encoded and decoded.
var t = T{X: 7, Y: 0, Z: 8}

the encoding of t sends only the 7 and 8. Because it's zero, the value of Y isn't even sent; there's no need to send a zero value.

The receiver could instead decode the value into this structure:

type U struct{ X, Y *int8 } // Note: pointers to int8s
var u U

and acquire a value of u with only X set (to the address of an int8 variable set to 7); the Z field is ignored - where would you put it? When decoding structs, fields are matched by name and compatible type, and only fields that exist in both are affected. This simple approach finesses the "optional field" problem: as the type T evolves by adding fields, out of date receivers will still function with the part of the type they recognize. Thus gobs provide the important result of optional fields - extensibility - without any additional mechanism or notation.

From integers we can build all the other types: bytes, strings, arrays, slices, maps, even floats. Floating-point values are represented by their IEEE 754 floating-point bit pattern, stored as an integer, which works fine as long as you know their type, which we always do. By the way, that integer is sent in byte-reversed order because common values of floating-point numbers, such as small integers, have a lot of zeros at the low end that we can avoid transmitting.

One nice feature of gobs that Go makes possible is that they allow you to define your own encoding by having your type satisfy the GobEncoder and GobDecoder interfaces, in a manner analogous to the JSON package's Marshaler and Unmarshaler and also to the Stringer interface from package fmt. This facility makes it possible to represent special features, enforce constraints, or hide secrets when you transmit data. See the documentation for details.

Types on the wire

The first time you send a given type, the gob package includes in the data stream a description of that type. In fact, what happens is that the encoder is used to encode, in the standard gob encoding format, an internal struct that describes the type and gives it a unique number. (Basic types, plus the layout of the type description structure, are predefined by the software for bootstrapping.) After the type is described, it can be referenced by its type number.

Thus when we send our first type T, the gob encoder sends a description of T and tags it with a type number, say 127. All values, including the first, are then prefixed by that number, so a stream of T values looks like:

("define type id" 127, definition of type T)(127, T value)(127, T value), ...

These type numbers make it possible to describe recursive types and send values of those types. Thus gobs can encode types such as trees:

type Node struct {
    Value       int
    Left, Right *Node
}

(It's an exercise for the reader to discover how the zero-defaulting rule makes this work, even though gobs don't represent pointers.)

With the type information, a gob stream is fully self-describing except for the set of bootstrap types, which is a well-defined starting point.

Compiling a machine

The first time you encode a value of a given type, the gob package builds a little interpreted machine specific to that data type. It uses reflection on the type to construct that machine, but once the machine is built it does not depend on reflection. The machine uses package unsafe and some trickery to convert the data into the encoded bytes at high speed. It could use reflection and avoid unsafe, but would be significantly slower. (A similar high-speed approach is taken by the protocol buffer support for Go, whose design was influenced by the implementation of gobs.) Subsequent values of the same type use the already-compiled machine, so they can be encoded right away.

Decoding is similar but harder. When you decode a value, the gob package holds a byte slice representing a value of a given encoder-defined type to decode, plus a Go value into which to decode it. The gob package builds a machine for that pair: the gob type sent on the wire crossed with the Go type provided for decoding. Once that decoding machine is built, though, it's again a reflectionless engine that uses unsafe methods to get maximum speed.

Use

There's a lot going on under the hood, but the result is an efficient, easy-to-use encoding system for transmitting data. Here's a complete example showing differing encoded and decoded types. Note how easy it is to send and receive values; all you need to do is present values and variables to the gob package and it does all the work.

package main

import (
    "bytes"
    "encoding/gob"
    "fmt"
    "log"
)

type P struct {
    X, Y, Z int
    Name    string
}

type Q struct {
    X, Y *int32
    Name string
}

func main() {
    // Initialize the encoder and decoder.  Normally enc and dec would be
    // bound to network connections and the encoder and decoder would
    // run in different processes.
    var network bytes.Buffer        // Stand-in for a network connection
    enc := gob.NewEncoder(&network) // Will write to network.
    dec := gob.NewDecoder(&network) // Will read from network.
    // Encode (send) the value.
    err := enc.Encode(P{3, 4, 5, "Pythagoras"})
    if err != nil {
        log.Fatal("encode error:", err)
    }
    // Decode (receive) the value.
    var q Q
    err = dec.Decode(&q)
    if err != nil {
        log.Fatal("decode error:", err)
    }
    fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
}

You can compile and run this example code in the Go Playground.

The rpc package builds on gobs to turn this encode/decode automation into transport for method calls across the network. That's a subject for another article.

Details

The gob package documentation, especially the file doc.go, expands on many of the details described here and includes a full worked example showing how the encoding represents data. If you are interested in the innards of the gob implementation, that's a good place to start.

C? Go? Cgo!

Cgo lets Go packages call C code. Given a Go source file written with some special features, cgo outputs Go and C files that can be combined into a single Go package.

To lead with an example, here's a Go package that provides two functions - Random and Seed - that wrap C's random and srandom functions.

package rand

/*
#include <stdlib.h>
*/
import "C"

func Random() int {
    return int(C.random())
}

func Seed(i int) {
    C.srandom(C.uint(i))
}

Let's look at what's happening here, starting with the import statement.

The rand package imports "C", but you'll find there's no such package in the standard Go library. That's because C is a "pseudo-package", a special name interpreted by cgo as a reference to C's name space.

The rand package contains four references to the C package: the calls to C.random and C.srandom, the conversion C.uint(i), and the import statement.

The Random function calls the standard C library's random function and returns the result. In C, random returns a value of the C type long, which cgo represents as the type C.long. It must be converted to a Go type before it can be used by Go code outside this package, using an ordinary Go type conversion:

func Random() int {
    return int(C.random())
}

Here's an equivalent function that uses a temporary variable to illustrate the type conversion more explicitly:

func Random() int {
    var r C.long = C.random()
    return int(r)
}

The Seed function does the reverse, in a way. It takes a regular Go int, converts it to the C unsigned int type, and passes it to the C function srandom.

func Seed(i int) {
    C.srandom(C.uint(i))
}

Note that cgo knows the unsigned int type as C.uint; see the cgo documentation for a complete list of these numeric type names.

The one detail of this example we haven't examined yet is the comment above the import statement.

/*
#include <stdlib.h>
*/
import "C"

Cgo recognizes this comment. Any lines starting with #cgo followed by a space character are removed; these become directives for cgo. The remaining lines are used as a header when compiling the C parts of the package. In this case those lines are just a single #include statement, but they can be almost any C code. The #cgo directives are used to provide flags for the compiler and linker when building the C parts of the package.

There is a limitation: if your program uses any //export directives, then the C code in the comment may only include declarations (extern int f();), not definitions (int f() { return 1; }). You can use //export directives to make Go functions accessible to C code.

The #cgo and //export directives are documented in the cgo documentation.

Strings and things

Unlike Go, C doesn't have an explicit string type. Strings in C are represented by a zero-terminated array of chars.

Conversion between Go and C strings is done with the C.CString, C.GoString, and C.GoStringN functions. These conversions make a copy of the string data.

This next example implements a Print function that writes a string to standard output using C's fputs function from the stdio library:

package print

// #include <stdio.h>
// #include <stdlib.h>
import "C"
import "unsafe"

func Print(s string) {
    cs := C.CString(s)
    C.fputs(cs, (*C.FILE)(C.stdout))
    C.free(unsafe.Pointer(cs))
}

Memory allocations made by C code are not known to Go's memory manager. When you create a C string with C.CString (or any C memory allocation) you must remember to free the memory when you're done with it by calling C.free.

The call to C.CString returns a pointer to the start of the char array, so before the function exits we convert it to an unsafe.Pointer and release the memory allocation with C.free. A common idiom in cgo programs is to defer the free immediately after allocating (especially when the code that follows is more complex than a single function call), as in this rewrite of Print:

func Print(s string) {
    cs := C.CString(s)
    defer C.free(unsafe.Pointer(cs))
    C.fputs(cs, (*C.FILE)(C.stdout))
}

Building cgo packages

To build cgo packages, just use " go build" or "go install " as usual. The go tool recognizes the special "C" import and automatically uses cgo for those files.

More cgo resources

The cgo command documentation has more detail about the C pseudo-package and the build process. The cgo examples in the Go tree demonstrate more advanced concepts.

For a simple, idiomatic example of a cgo-based package, see Russ Cox's gosqlite. Also, the Go Project Dashboard lists several other cgo packages.

Finally, if you're curious as to how all this works internally, take a look at the introductory comment of the runtime package's cgocall.c.

Go becomes more stable

The Go project is moving fast. As we learn more about Go we are compelled to change our tools, libraries, and occasionally even the language itself. We permit backward-incompatible changes so that we can learn from, rather than immortalize, our mistakes. We believe flexibility at this stage of Go’s development is essential to the project’s development and, ultimately, its longevity.

Since Go's launch we have been making releases approximately once a week. Accompanying each release are notes describing what changed, with any backwards-incompatible changes signposted. Questions I hear often are "Is Go stable? How can I be sure that I won’t have to update my Go code every week?" The answer to those questions are now "Yes," and "You won't."

With this week’s release we’re introducing a new release tagging scheme. We intend to continue with our weekly releases, but have renamed the existing tags from release to weekly. The release tag will now be applied to one hand-picked stable release each month or two. This more relaxed release schedule should make life easier for the average Go programmer.

Users will still need to update their code periodically (this is the cost of using a young language) but with less frequency. An additional benefit is that by tagging stable releases less often we can put more effort into automating updates. To this end we have introduced gofix, a tool that will help you update your code.

The revision formerly tagged release.2011-03-07.1 (now weekly.2011-03-07.1) has been nominated our first stable release, and has been given the tag release.r56. As we tag each stable release we will post an announcement to the new golang-announce mailing list. (Subscribe now!)

What’s the upshot of all this? You can continue to keep your Go installation updated using hg update release, but now you should only need to update when we tag a new stable release. If you wish to stay at the leading edge, you should switch to the weekly tag with hg update weekly.

Happy coding!