Black Lives Matter. Support the Equal Justice Initiative.

The Go Blog

Go 1.15 is released

Alex Rakoczy
11 August 2020

Today the Go team is very happy to announce the release of Go 1.15. You can get it from the download page.

Some of the highlights include:

For the complete list of changes and more information about the improvements above, see the Go 1.15 release notes.

We want to thank everyone who contributed to this release by writing code, filing bugs, providing feedback, and/or testing the beta and release candidates. Your contributions and diligence helped to ensure that Go 1.15 is as stable as possible. That said, if you notice any problems, please file an issue.

We hope you enjoy the new release!

Keeping Your Modules Compatible

Jean de Klerk and Jonathan Amsterdam
7 July 2020

Introduction

This post is part 5 in a series.

Your modules will evolve over time as you add new features, change behaviors, and reconsider parts of the module's public surface. As discussed in Go Modules: v2 and Beyond, breaking changes to a v1+ module must happen as part of a major version bump (or by adopting a new module path).

However, releasing a new major version is hard on your users. They have to find the new version, learn a new API, and change their code. And some users may never update, meaning you have to maintain two versions for your code forever. So it is usually better to change your existing package in a compatible way.

In this post, we'll explore some techniques for introducing non-breaking changes. The common theme is: add, don’t change or remove. We’ll also talk about how to design your API for compatibility from the outset.

Adding to a function

Often, breaking changes come in the form of new arguments to a function. We’ll describe some ways to deal with this sort of change, but first let’s look at a technique that doesn’t work.

When adding new arguments with sensible defaults, it’s tempting to add them as a variadic parameter. To extend the function

func Run(name string)

with an additional size argument which defaults to zero, one might propose

func Run(name string, size ...int)

on the grounds that all existing call sites will continue to work. While that is true, other uses of Run could break, like this one:

package mypkg
var runner func(string) = yourpkg.Run

The original Run function works here because its type is func(string), but the new Run function’s type is func(string, ...int), so the assignment fails at compile time.

This example illustrates that call compatibility is not enough for backward compatibility. There is, in fact, no backward-compatible change you can make to a function’s signature.

Instead of changing a function’s signature, add a new function. As an example, after the context package was introduced, it became common practice to pass a context.Context as the first argument to a function. However, stable APIs could not change an exported function to accept a context.Context because it would break all uses of that function.

Instead, new functions were added. For example, the database/sql package's Query method’s signature was (and still is)

func (db *DB) Query(query string, args ...interface{}) (*Rows, error)

When the context package was created, the Go team added a new method to database/sql:

func (db *DB) QueryContext(ctx context.Context, query string, args ...interface{}) (*Rows, error)

To avoid copying code, the old method calls the new one:

func (db *DB) Query(query string, args ...interface{}) (*Rows, error) {
    return db.QueryContext(context.Background(), query, args...)
}

Adding a method allows users to migrate to the new API at their own pace. Since the methods read similarly and sort together, and Context is in the name of the new method, this extension of the database/sql API did not degrade readability or comprehension of the package.

If you anticipate that a function may need more arguments in the future, you can plan ahead by making optional arguments a part of the function’s signature. The simplest way to do that is to add a single struct argument, as the crypto/tls.Dial function does:

func Dial(network, addr string, config *Config) (*Conn, error)

The TLS handshake conducted by Dial requires a network and address, but it has many other parameters with reasonable defaults. Passing a nil for config uses those defaults; passing a Config struct with some fields set will override the defaults for those fields. In the future, adding a new TLS configuration parameter only requires a new field on the Config struct, a change that is backward-compatible (almost always—see "Maintaining struct compatibility" below).

Sometimes the techniques of adding a new function and adding options can be combined by making the options struct a method receiver. Consider the evolution of the net package’s ability to listen at a network address. Prior to Go 1.11, the net package provided only a Listen function with the signature

func Listen(network, address string) (Listener, error)

For Go 1.11, two features were added to net listening: passing a context, and allowing the caller to provide a “control function” to adjust the raw connection after creation but before binding. The result could have been a new function that took a context, network, address and control function. Instead, the package authors added a ListenConfig struct in anticipation that more options might be needed someday. And rather than define a new top-level function with a cumbersome name, they added a Listen method to ListenConfig:

type ListenConfig struct {
    Control func(network, address string, c syscall.RawConn) error
}

func (*ListenConfig) Listen(ctx context.Context, network, address string) (Listener, error)

Another way to provide new options in the future is the “Option types” pattern, where options are passed as variadic arguments, and each option is a function that changes the state of the value being constructed. They are described in more detail by Rob Pike's post Self-referential functions and the design of options. One widely used example is google.golang.org/grpc's DialOption.

Option types fulfill the same role as struct options in function arguments: they are an extensible way to pass behavior-modifying configuration. Deciding which to choose is largely a matter of style. Consider this simple usage of gRPC's DialOption option type:

grpc.Dial("some-target",
  grpc.WithAuthority("some-authority"),
  grpc.WithMaxDelay(time.Second),
  grpc.WithBlock())

This could also have been implemented as a struct option:

notgrpc.Dial("some-target", &notgrpc.Options{
  Authority: "some-authority",
  MaxDelay:  time.Minute,
  Block:     true,
})

Functional options have some downsides: they require writing the package name before the option for each call; they increase the size of the package namespace; and it's unclear what the behavior should be if the same option is provided twice. On the other hand, functions which take option structs require a parameter which might almost always be nil, which some find unattractive. And when a type’s zero value has a valid meaning, it is clumsy to specify that the option should have its default value, typically requiring a pointer or an additional boolean field.

Either one is a reasonable choice for ensuring future extensibility of your module's public API.

Working with interfaces

Sometimes, new features require changes to publicly-exposed interfaces: for example, an interface needs to be extended with new methods. Directly adding to an interface is a breaking change, though—how, then, can we support new methods on a publicly-exposed interface?

The basic idea is to define a new interface with the new method, and then wherever the old interface is used, dynamically check whether the provided type is the older type or the newer type.

Let's illustrate this with an example from the archive/tar package. tar.NewReader accepts an io.Reader, but over time the Go team realized that it would be more efficient to skip from one file header to the next if you could call Seek. But, they could not add a Seek method to io.Reader: that would break all implementers of io.Reader.

Another ruled-out option was to change tar.NewReader to accept io.ReadSeeker rather than io.Reader, since it supports both io.Reader methods and Seek (by way of io.Seeker). But, as we saw above, changing a function signature is also a breaking change.

So, they decided to keep tar.NewReader signature unchanged, but type check for (and support) io.Seeker in tar.Reader methods:

package tar

type Reader struct {
  r io.Reader
}

func NewReader(r io.Reader) *Reader {
  return &Reader{r: r}
}

func (r *Reader) Read(b []byte) (int, error) {
  if rs, ok := r.r.(io.Seeker); ok {
    // Use more efficient rs.Seek.
  }
  // Use less efficient r.r.Read.
}

(See reader.go for the actual code.)

When you run into a case where you want to add a method to an existing interface, you may be able to follow this strategy. Start by creating a new interface with your new method, or identify an existing interface with the new method. Next, identify the relevant functions that need to support it, type check for the second interface, and add code that uses it.

This strategy only works when the old interface without the new method can still be supported, limiting the future extensibility of your module.

Where possible, it is better to avoid this class of problem entirely. When designing constructors, for example, prefer to return concrete types. Working with concrete types allows you to add methods in the future without breaking users, unlike interfaces. That property allows your module to be extended more easily in the future.

Tip: if you do need to use an interface but don't intend for users to implement it, you can add an unexported method. This prevents types defined outside your package from satisfying your interface without embedding, freeing you to add methods later without breaking user implementations. For example, see testing.TB's private() function.

type TB interface {
    Error(args ...interface{})
    Errorf(format string, args ...interface{})
    // ...

    // A private method to prevent users implementing the
    // interface and so future additions to it will not
    // violate Go 1 compatibility.
    private()
}

This topic is also explored in more detail in Jonathan Amsterdam's "Detecting Incompatible API Changes" talk (video, slides).

Add configuration methods

So far we've talked about overt breaking changes, where changing a type or a function would cause users' code to stop compiling. However, behavior changes can also break users, even if user code continues to compile. For example, many users expect json.Decoder to ignore fields in the JSON that are not in the argument struct. When the Go team wanted to return an error in that case, they had to be careful. Doing so without an opt-in mechanism would mean that the many users relying on those methods might start receiving errors where they hadn’t before.

So, rather than changing the behavior for all users, they added a configuration method to the Decoder struct: Decoder.DisallowUnknownFields. Calling this method opts a user in to the new behavior, but not doing so preserves the old behavior for existing users.

Maintaining struct compatibility

We saw above that any change to a function’s signature is a breaking change. The situation is much better with structs. If you have an exported struct type, you can almost always add a field or remove an unexported field without breaking compatibility. When adding a field, make sure that its zero value is meaningful and preserves the old behavior, so that existing code that doesn’t set the field continues to work.

Recall that the authors of the net package added ListenConfig in Go 1.11 because they thought more options might be forthcoming. Turns out they were right. In Go 1.13, the KeepAlive field was added to allow for disabling keep-alive or changing its period. The default value of zero preserves the original behavior of enabling keep-alive with a default period.

There is one subtle way a new field can break user code unexpectedly. If all the field types in a struct are comparable—meaning values of those types can be compared with == and != and used as a map key—then the overall struct type is comparable too. In this case, adding a new field of uncomparable type will make the overall struct type non-comparable, breaking any code that compares values of that struct type.

To keep a struct comparable, don’t add non-comparable fields to it. You can write a test for that, or rely on the upcoming gorelease tool to catch it.

To prevent comparison in the first place, make sure the struct has a non-comparable field. It may have one already—no slice, map or function type is comparable—but if not, one can be added like so:

type Point struct {
        _ [0]func()
        X int
        Y int
}

The func() type is not comparable, and the zero-length array takes up no space. We can define a type to clarify our intent:

type doNotCompare [0]func()

type Point struct {
        doNotCompare
        X int
        Y int
}

Should you use doNotCompare in your structs? If you’ve defined the struct to be used as a pointer—that is, it has pointer methods and perhaps a NewXXX constructor function that returns a pointer—then adding a doNotCompare field is probably overkill. Users of a pointer type understand that each value of the type is distinct: that if they want to compare two values, they should compare the pointers.

If you are defining a struct intended to be used as a value directly, like our Point example, then quite often you want it to be comparable. In the uncommon case that you have a value struct that you don’t want compared, then adding a doNotCompare field will give you the freedom to change the struct later without having to worry about breaking comparisons. On the downside, the type won’t be usable as a map key.

Conclusion

When planning an API from scratch, consider carefully how extensible the API will be to new changes in the future. And when you do need to add new features, remember the rule: add, don't change or remove, keeping in mind the exceptions—interfaces, function arguments, and return values can't be added in backwards-compatible ways.

If you need to dramatically change an API, or if an API begins to lose its focus as more features are added, then it may be time for a new major version. But most of the time, making a backwards-compatible change is easy and avoids causing pain for your users.

The Next Step for Generics

Ian Lance Taylor and Robert Griesemer
16 June 2020

Introduction

It’s been almost a year since we last wrote about the possibility of adding generics to Go. It’s time for an update.

Updated design

We’ve been continuing to refine the generics design draft. We’ve written a type checker for it: a program that can parse Go code that uses generics as described in the design draft and report any type errors. We’ve written example code. And we’ve collected feedback from many, many people—thanks for providing it!

Based on what we’ve learned, we’re releasing an updated design draft. The biggest change is that we are dropping the idea of contracts. The difference between contracts and interface types was confusing, so we’re eliminating that difference. Type parameters are now constrained by interface types. Interface types are now permitted to include type lists, though only when used as constraints; in the previous design draft type lists were a feature of contracts. More complex cases will use a parameterized interface type.

We hope that people will find this design draft simpler and easier to understand.

Experimentation tool

To help decide how to further refine the design draft, we are releasing a translation tool. This is a tool that permits people to type check and run code written using the version of generics described in the design draft. It works by translating generic code into ordinary Go code. This translation process imposes some limitations, but we hope that it will be good enough for people to get a feel for what generic Go code might look like. The real implementation of generics, if they are accepted into the language, will work differently. (We have only just begun to sketch out what a direct compiler implementation would look like.)

The tool is available on a variant of the Go playground at https://go2goplay.golang.org. This playground works just like the usual Go playground, but it supports generic code.

You can also build and use the tool yourself. It is available in a branch of the master Go repo. Follow the instructions on installing Go from source. Where those instructions direct you to check out the latest release tag, instead run git checkout dev.go2go. Then build the Go toolchain as directed.

The translation tool is documented in README.go2go.

Next steps

We hope that the tool will give the Go community a chance to experiment with generics. There are two main things that we hope to learn.

First, does generic code make sense? Does it feel like Go? What surprises do people encounter? Are the error messages useful?

Second, we know that many people have said that Go needs generics, but we don’t necessarily know exactly what that means. Does this draft design address the problem in a useful way? If there is a problem that makes you think “I could solve this if Go had generics,” can you solve the problem when using this tool?

We will use the feedback we gather from the Go community to decide how to move forward. If the draft design is well received and doesn’t need significant changes, the next step would be a formal language change proposal. To set expectations, if everybody is completely happy with the design draft and it does not require any further adjustments, the earliest that generics could be added to Go would be the Go 1.17 release, scheduled for August 2021. In reality, of course, there may be unforeseen problems, so this is an optimistic timeline; we can’t make any definite prediction.

Feedback

The best way to provide feedback for the language changes will be on the mailing list golang-nuts@googlegroups.com. Mailing lists are imperfect, but they seem like our best option for initial discussion. When writing about the design draft, please put [generics] at the start of the Subject line and to start different threads for different specific topics.

If you find bugs in the generics type checker or the translation tool, they should be filed in the standard Go issue tracker at https://golang.org/issue. Please start the issue title with cmd/go2go:. Note that the issue tracker is not the best place to discuss changes to the language, because it does not provide threading and it is not well suited to lengthy conversations.

We look forward to your feedback.

Acknowledgements

We’re not finished, but we’ve come a long way. We would not be here without a lot of help.

We’d like to thank Philip Wadler and his collaborators for thinking formally about generics in Go and helping us clarify the theoretical aspects of the design. Their paper Featherweight Go analyzes generics in a restricted version of Go, and they have developed a prototype on GitHub.

We would also like to thank the people who provided detailed feedback on an earlier version of the design draft.

And last but definitely not least, we’d like to thank many people on the Go team, many contributors to the Go issue tracker, and everybody else who shared ideas and feedback on earlier design drafts. We read all of it, and we’re grateful. We wouldn’t be here without you.

Pkg.go.dev is open source!

Julie Qiu
15 June 2020

We’re excited to announce that the codebase for pkg.go.dev is now open source.

The repository lives at go.googlesource.com/pkgsite and is mirrored to github.com/golang/pkgsite. We will continue using the Go issue tracker to track feedback related to pkg.go.dev.

Contributing

If you are interested in contributing to any issues related to pkg.go.dev, check out our contribution guidelines. We also encourage you to continue filing issues if you run into problems or have feedback.

What’s Next

We really appreciate all the feedback we’ve received so far. It has been a big help in shaping our roadmap for the coming year. Now that pkg.go.dev is open source, here’s what we’ll be working on next:

  • We have some design changes planned for pkg.go.dev, to address UX feedback that we have received. You can expect a more cohesive search and navigation experience. We plan to share these designs for feedback once they are ready.

  • We know that there are features available on godoc.org that users want to see on pkg.go.dev. We’ve been keeping track of them on Go issue #39144, and will prioritize adding them in the next few months. We also plan to continue improving our license detection algorithm based on feedback.

  • We’ll be improving our search experience based on feedback in Go issue #37810, to make it easier for users to find the dependencies they are looking for and make better decisions around which ones to import.

Thanks for being patient with us in the process of open sourcing pkg.go.dev. We’re looking forward to receiving your contributions and working with you on the future of the project.

The VS Code Go extension joins the Go project

The Go team
9 June 2020

When the Go project began, “an overarching goal was that Go do more to help the working programmer by enabling tooling, automating mundane tasks such as code formatting, and removing obstacles to working on large code bases” (Go FAQ). Today, more than a decade later, we continue to be guided by that same goal, especially as it pertains to the programmer’s most critical tool: their editor.

Throughout the past decade, Go developers have relied on a variety of editors and dozens of independently authored tools and plugins. Much of Go’s early success can be attributed to the fantastic development tools created by the Go community. The VS Code extension for Go, built using many of these tools, is now used by 41 percent of Go developers (Go developer survey).

As the VS Code Go extension grows in popularity and as the ecosystem expands, it requires more maintenance and support. Over the past few years, the Go team has collaborated with the VS Code team to help the Go extension maintainers. The Go team also began a new initiative to improve the tools powering all Go editor extensions, with a focus on supporting the Language Server Protocol with gopls and the Debug Adapter Protocol with Delve.

Through this collaborative work between the VS Code and Go teams, we realized that the Go team is uniquely positioned to evolve the Go development experience alongside the Go language.

As a result, we’re happy to announce the next phase in the Go team’s partnership with the VS Code team: The VS Code extension for Go is officially joining the Go project. With this come two critical changes:

  1. The publisher of the plugin is shifting from "Microsoft" to "Go Team at Google".
  2. The project’s repository is moving to join the rest of the Go project at https://github.com/golang/vscode-go.

We cannot overstate our gratitude to those who have helped build and maintain this beloved extension. We know that innovative ideas and features come from you, our users. The Go team’s primary aim as owners of the extension is to reduce the burden of maintenance work on the Go community. We’ll make sure the builds stay green, the issues get triaged, and the docs get updated. Go team members will keep contributors abreast of relevant language changes, and we’ll smooth the rough edges between the extension’s different dependencies.

Please continue to share your thoughts with us by filing issues and making contributions to the project. The process for contributing will now be the same as for the rest of the Go project. Go team members will offer general help in the #vscode channel on Gophers Slack, and we’ve also created a #vscode-dev channel to discuss issues and brainstorm ideas with contributors.

We’re excited about this new step forward, and we hope you are too. By maintaining a major Go editor extension, as well as the Go tooling and language, the Go team will be able to provide all Go users, regardless of their editor, a more cohesive and refined development experience.

As always, our goal remains the same: Every user should have an excellent experience writing Go code.

See the accompanying post from the Visual Studio Code team.

See the index for more articles.