The Go Blog

Go 2, here we come!

29 November 2018

Background

At GopherCon 2017, Russ Cox officially started the thought process on the next big version of Go with his talk The Future of Go (blog post). We have called this future language informally Go 2, even though we understand now that it will arrive in incremental steps rather than with a big bang and a single major release. Still, Go 2 is a useful moniker, if only to have a way to talk about that future language, so let’s keep using it for now.

A major difference between Go 1 and Go 2 is who is going to influence the design and how decisions are made. Go 1 was a small team effort with modest outside influence; Go 2 will be much more community-driven. After almost 10 years of exposure, we have learned a lot about the language and libraries that we didn’t know in the beginning, and that was only possible through feedback from the Go community.

In 2015 we introduced the proposal process to gather a specific kind of feedback: proposals for language and library changes. A committee composed of senior Go team members has been reviewing, categorizing, and deciding on incoming proposals on a regular basis. That has worked pretty well, but as part of that process we have ignored all proposals that are not backward-compatible, simply labeling them Go 2 instead. In 2017 we also stopped making any kind of incremental backward-compatible language changes, however small, in favor of a more comprehensive plan that takes the bigger picture of Go 2 into account.

It is now time to act on the Go 2 proposals, but to do this we first need a plan.

Status

At the time of writing, there are around 120 open issues labeled Go 2 proposal. Each of them proposes a significant library or language change, often one that does not satisfy the existing Go 1 compatibility guarantee. Ian Lance Taylor and I have been working through these proposals and categorized them (Go2Cleanup, NeedsDecision, etc.) to get an idea of what’s there and to make it easier to proceed with them. We also merged related proposals and closed the ones which seemed clearly out of the scope of Go, or were otherwise unactionable.

Ideas from the remaining proposals will likely influence Go 2’s libraries and languages. Two major themes have emerged early on: support for better error handling, and generics. Draft designs for these two areas have been published at this year’s GopherCon, and more exploration is needed.

But what about the rest? We are constrained by the fact that we now have millions of Go programmers and a large body of Go code, and we need to bring it all along, lest we risk a split ecosystem. That means we cannot make many changes, and the changes we are going to make need to be chosen carefully. To make progress, we are implementing a new proposal evaluation process for these significant potential changes.

Proposal evaluation process

The purpose of the proposal evaluation process is to collect feedback on a small number of select proposals such that a final decision can be made. The process runs more or less in parallel to a release cycle and consists of the following steps:

1. Proposal selection. The Go team selects a small number of Go 2 proposals that seem worth considering for acceptance, without making a final decision. See below for more on the selection criteria.

2. Proposal feedback. The Go team sends out an announcement listing the selected proposals. The announcement explains to the community the tentative intent to move forward with the selected proposals and to collect feedback for each of them. This gives the community a chance to make suggestions and express concerns.

3. Implementation. Based on that feedback, the proposals are implemented. The target for these significant language and library changes is to have them ready to submit on day 1 of an upcoming release cycle.

4. Implementation feedback. During the development cycle, the Go team and community have a chance to experiment with the new features and collect further feedback.

5. Launch decision. At the end of the three month development cycle (just when starting the three month repo freeze before a release), and based on the experience and feedback gathered during the release cycle, the Go team makes the final decision about whether to ship each change. This provides an opportunity to consider whether the change has delivered the expected benefits or created any unexpected costs. Once shipped, the changes become part of the language and libraries. Excluded proposals may go back to the drawing board or may be declined for good.

With two rounds of feedback, this process is slanted towards declining proposals, which will hopefully prevent feature creep and help with keeping the language small and clean.

We can’t go through this process for each of the open Go 2 proposals, there are simply too many of them. That’s where the selection criteria come into play.

Proposal selection criteria

A proposal must at the very least:

1. address an important issue for many people,

2. have minimal impact on everybody else, and

3. come with a clear and well-understood solution.

Requirement 1 ensures that any changes we make help as many Go developers as possible (make their code more robust, easier to write, more likely to be correct, and so on), while requirement 2 ensures we are careful to hurt as few developers as possible, whether by breaking their programs or causing other churn. As a rule of thumb, we should aim to help at least ten times as many developers as we hurt with a given change. Changes that don't affect real Go usage are a net zero benefit put up against a significant implementation cost and should be avoided.

Without requirement 3 we don’t have an implementation of the proposal. For instance, we believe that some form of genericity might solve an important issue for a lot of people, but we don’t yet have a clear and well-understood solution. That’s fine, it just means that the proposal needs to go back to the drawing board before it can be considered.

Proposals

We feel that this is a good plan that should serve us well but it is important to understand that this is only a starting point. As the process is used we will discover the ways in which it fails to work well and we will refine it as needed. The critical part is that until we use it in practice we won't know how to improve it.

A safe place to start is with a small number of backward-compatible language proposals. We haven’t done language changes for a long time, so this gets us back into that mode. Also, the changes won’t require us worrying about breaking existing code, and thus they serve as a perfect trial balloon.

With all that said, we propose the following selection of Go 2 proposals for the Go 1.13 release (step 1 in the proposal evaluation process):

1. #20706 General Unicode identifiers based on Unicode TR31: This addresses an important issue for Go programmers using non-Western alphabets and should have little if any impact on anyone else. There are normalization questions which we need to answer and where community feedback will be important, but after that the implementation path is well understood. Note that identifier export rules will not be affected by this.

2. #19308, #28493 Binary integer literals and support for_ in number literals: These are relatively minor changes that seem hugely popular among many programmers. They may not quite reach the threshold of solving an “important issue” (hexadecimal numbers have worked well so far) but they bring Go up to par with most other languages in this respect and relieve a pain point for some programmers. They have minimal impact on others who don’t care about binary integer literals or number formatting, and the implementation is well understood.

3. #19113 Permit signed integers as shift counts: An estimated 38% of all non-constant shifts require an (artificial) uint conversion (see the issue for a more detailed break-down). This proposal will clean up a lot of code, get shift expressions better in sync with index expressions and the built-in functions cap and len. It will mostly have a positive impact on code. The implementation is well understood.

Next steps

With this blog post we have executed the first step and started the second step of the proposal evaluation process. It’s now up to you, the Go community, to provide feedback on the issues listed above.

For each proposal for which we have clear and approving feedback, we will move forward with the implementation (step 3 in the process). Because we want the changes implemented on the first day of the next release cycle (tentatively Feb. 1, 2019) we may start the implementation a bit early this time to leave time for two full months of feedback (Dec. 2018, Jan. 2019).

For the 3-month development cycle (Feb. to May 2019) the chosen features are implemented and available at tip and everybody will have a chance to gather experience with them. This provides another opportunity for feedback (step 4 in the process).

Finally, shortly after the repo freeze (May 1, 2019), the Go team makes the final decision whether to keep the new features for good (and include them in the Go 1 compatibility guarantee), or whether to abandon them (final step in the process).

(Since there is a real chance that a feature may need to be removed just when we freeze the repo, the implementation will need to be such that the feature can be disabled without destabilizing the rest of the system. For language changes that may mean that all feature-related code is guarded by an internal flag.)

This will be the first time that we have followed this process, hence the repo freeze will also be a good moment to reflect on the process and to adjust it if necessary. Let’s see how it goes.

Happy evaluating!

By Robert Griesemer

Nine years of Go

10 November 2018

Introduction

Today marks the ninth anniversary of the day we open-sourced our initial sketch of Go. On each anniversary we like to take time to reflect on what has happened over the past year. The past 12 months have been a breakout year for the Go language and community.

Go Love & Adoption

Thanks to all of you, 2018 was an amazing year for Go! In multiple industry surveys Gophers expressed how happy they were using Go, and many non-Go developers indicated they intended to learn Go before any other language.

In Stack Overflow’s 2018 Developer Survey, Go retained its coveted spot in both the top 5 most loved and top 5 most wanted languages. People who use Go love it, and people who aren’t using Go want to.

In ActiveState’s 2018 Developer Survey, Go topped the charts with 36% of users responding they were “Extremely Satisfied” using Go and 61% responding “Very Satisfied” or better.

JetBrains’s 2018 Developer Survey awarded Go the “Most promising language” with 12% of respondents using Go today and 16% intending to use Go in the future.

In HackerRank’s 2018 Developer Survey, 38% of developers responded that they were intending to learn Go next.

We are excited about all of our new gophers and actively working to improve our educational and community resources.

Go Community

It’s hard to believe that it’s only been five years since the first Go conferences and Go meetups. We’ve seen major growth in this area of community leadership over the last year. There are now over 20 Go conferences and over 300 Go-related meetups spanning the globe.

Thanks to the hard work put into these many conferences and meetups, there have been hundreds of great talks this year. Here are a few of our favorite talks specifically discussing the growth of our community and how we can better support gophers worldwide.

On that theme, this year we also revised our code of conduct to better support inclusivity in the Go community.

The Go community is truly global. At GopherCon Europe in Iceland this past summer, gophers literally spanned the gap between the continents.

(Photo by Winter Francia.)

Go 2

After five years of experience with Go 1, we’ve started looking at what we should change about Go to better support programming at scale.

Last spring, we published a draft design for Go modules, which provide an integrated mechanism for versioning and package distribution. The most recent Go release, Go 1.11, included preliminary support for modules.

Last summer we published early draft designs for how Go 2 might better support error values, error handling, and generic programming.

We are excited about refining these designs with the community’s help as we work toward Go 2.

Go Contributors

The Go project has been increasing in the number of contributions from the community for several years. The project hit a major milestone in mid-2018 when, for the first time, we had more contributions coming from the community than the Go team.

Thank You

On a personal note, from the entire Go team, we want to sincerely thank all of you. We feel privileged to be able to work on the Go project and are grateful to the many gophers around the world who have joined us.

We are especially thankful for the thousands of volunteers who help through mentorship, organizing, contributing, and supporting your fellow gophers. You have made Go what it is today.

By Steve Francia

Participate in the 2018 Go User Survey

8 November 2018

The Go project wants to hear from you, the Go community!

In 2017 & 2016, thousands of you helped the project by lending your voice via the Go user survey. In October, hundreds of companies helped us understand how enterprises are using Go by taking the Go company questionnaire. These surveys and questionnaires have played an enormous role in driving changes to our language and community, from our new code of conduct, to our latest release Go 1.11.

Today we are conducting the 2018 Go user survey. We’d like to hear from all Go users in order to help us to create the best programming language that fits the needs and desires of the people closest to it. Please help us shape the future of Go by participating in the 15-minute 2018 Go user survey by November 30th: Go User Survey 2018.

This survey is confidential and anonymous. For more information, please refer to Google’s privacy policy here.

Spread the word!

We need as many Gophers as possible to participate in this survey to help us better understand our global user base. We’d be grateful if you would spread the word by sharing this post on your social network feeds, around the office, at meetups, and in other communities.

By Ran Tao, Steve Francia

Announcing App Engine’s New Go 1.11 Runtime

16 October 2018

App Engine launched experimental support for Go in 2011. In the subsequent years, the Go community has grown significantly and has settled on idiomatic patterns for cloud-based applications. Today, Google Cloud is announcing a new Go 1.11 runtime for the App Engine standard environment that provides all the power of App Engine—things like paying only for what you use, automatic scaling, and managed infrastructure—while supporting idiomatic Go.

Starting with Go 1.11, Go on App Engine has no limits on application structure, supported packages, context.Context values, or HTTP clients. Write your Go application however you prefer, add an app.yaml file, and your app is ready to deploy on App Engine. Specifying Dependencies describes how the new runtime supports vendoring and modules (experimental) for dependency management.

Along with Cloud Functions support for Go (more on that in a future post), App Engine provides a compelling way to run Go code on Google Cloud Platform (GCP) with no concern for the underlying infrastructure.

Let’s take a look at creating a small application for App Engine. For the example here, we assume a GOPATH-based workflow, although Go modules have experimental support as well.

First, you create the application in your GOPATH:

// This server can run on App Engine.
package main

import (
    "fmt"
    "log"
    "net/http"
    "os"
)

func main() {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    http.HandleFunc("/", hello)

    log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}

func hello(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Hello, 世界"))
}

The code contains an idiomatic setup for a small HTTP server that responds with “Hello, 世界.” If you have previous App Engine experience, you’ll notice the absence of any call to appengine.Main(), which is now entirely optional. Furthermore, the application code is completely portable—there are no ties to the infrastructure that your application is deployed on.

If you need to use external dependencies, you can add those dependencies to a vendor directory or to a go.mod file, both of which the new runtime supports.

With the application code complete, create an app.yaml file to specify the runtime:

runtime: go111

Finally, set your machine up with a Google Cloud Platform account:

With all the setup complete, you can deploy using one command:

gcloud app deploy

We think Go developers will find the new Go 1.11 runtime for App Engine an exciting addition to the available options to run Go applications. There is a free tier. Check out the getting started guide or the migration guide and deploy an app to the new runtime today!

By Eno Compton and Tyler Bui-Palsulich

Compile-time Dependency Injection With Go Cloud's Wire

9 October 2018

Overview

The Go team recently announced the open source project Go Cloud, with portable Cloud APIs and tools for open cloud development. This post goes into more detail about Wire, a dependency injection tool used in Go Cloud.

What problem does Wire solve?

Dependency injection is a standard technique for producing flexible and loosely coupled code, by explicitly providing components with all of the dependencies they need to work. In Go, this often takes the form of passing dependencies to constructors:

// NewUserStore returns a UserStore that uses cfg and db as dependencies.
func NewUserStore(cfg *Config, db *mysql.DB) (*UserStore, error) {...}

This technique works great at small scale, but larger applications can have a complex graph of dependencies, resulting in a big block of initialization code that's order-dependent but otherwise not very interesting. It's often hard to break up this code cleanly, especially because some dependencies are used multiple times. Replacing one implementation of a service with another can be painful because it involves modifying the dependency graph by adding a whole new set of dependencies (and their dependencies...), and removing unused old ones. In practice, making changes to initialization code in applications with large dependency graphs is tedious and slow.

Dependency injection tools like Wire aim to simplify the management of initialization code. You describe your services and their dependencies, either as code or as configuration, then Wire processes the resulting graph to figure out ordering and how to pass each service what it needs. Make changes to an application's dependencies by changing a function signature or adding or removing an initializer, and then let Wire do the tedious work of generating initialization code for the entire dependency graph.

Why is this part of Go Cloud?

Go Cloud's goal is to make it easier to write portable Cloud applications by providing idiomatic Go APIs for useful Cloud services. For example, blob.Bucket provides a storage API with implementations for Amazon's S3 and Google Cloud Storage (GCS); applications written using blob.Bucket can swap implementations without changing their application logic. However, the initialization code is inherently provider-specific, and each provider has a different set of dependencies.

For example, constructing a GCS blob.Bucket requires a gcp.HTTPClient, which eventually requires google.Credentials, while constructing one for S3 requires an aws.Config, which eventually requires AWS credentials. Thus, updating an application to use a different blob.Bucket implementation involves exactly the kind of tedious update to the dependency graph that we described above. The driving use case for Wire is to make it easy to swap implementations of Go Cloud portable APIs, but it's also a general-purpose tool for dependency injection.

Hasn't this been done already?

There are a number of dependency injection frameworks out there. For Go, Uber's dig and Facebook's inject both use reflection to do runtime dependency injection. Wire was primarily inspired by Java's Dagger 2, and uses code generation rather than reflection or service locators.

We think this approach has several advantages:

  • Runtime dependency injection can be hard to follow and debug when the dependency graph gets complex. Using code generation means that the initialization code that's executed at runtime is regular, idiomatic Go code that's easy to understand and debug. Nothing is obfuscated by an intervening framework doing "magic". In particular, problems like forgetting a dependency become compile-time errors, not run-time errors.
  • Unlike service locators, there's no need to make up arbitrary names or keys to register services. Wire uses Go types to connect components with their dependencies.
  • It's easier to avoid dependency bloat. Wire's generated code will only import the dependencies you need, so your binary won't have unused imports. Runtime dependency injectors can't identify unused dependencies until runtime.
  • Wire's dependency graph is knowable statically, which provides opportunities for tooling and visualization.

How does it work?

Wire has two basic concepts: providers and injectors.

Providers are ordinary Go functions that "provide" values given their dependencies, which are described simply as parameters to the function. Here's some sample code that defines three providers:

// NewUserStore is the same function we saw above; it is a provider for UserStore,
// with dependencies on *Config and *mysql.DB.
func NewUserStore(cfg *Config, db *mysql.DB) (*UserStore, error) {...}

// NewDefaultConfig is a provider for *Config, with no dependencies.
func NewDefaultConfig() *Config {...}

// NewDB is a provider for *mysql.DB based on some connection info.
func NewDB(info *ConnectionInfo) (*mysql.DB, error) {...}

Providers that are commonly used together can be grouped into ProviderSets. For example, it's common to use a default *Config when creating a *UserStore, so we can group NewUserStore and NewDefaultConfig in a ProviderSet:

var UserStoreSet = wire.ProviderSet(NewUserStore, NewDefaultConfig)

Injectors are generated functions that call providers in dependency order. You write the injector's signature, including any needed inputs as arguments, and insert a call to wire.Build with the list of providers or provider sets that are needed to construct the end result:

func initUserStore() (*UserStore, error) {
    // We're going to get an error, because NewDB requires a *ConnectionInfo
    // and we didn't provide one.
    wire.Build(UserStoreSet, NewDB)
    return nil, nil  // These return values are ignored.
}

Now we run go generate to execute wire:

$ go generate
wire.go:2:10: inject initUserStore: no provider found for ConnectionInfo (required by provider of *mysql.DB)
wire: generate failed

Oops! We didn't include a ConnectionInfo or tell Wire how to build one. Wire helpfully tells us the line number and types involved. We can either add a provider for it to wire.Build, or add it as an argument:

func initUserStore(info ConnectionInfo) (*UserStore, error) {
    wire.Build(UserStoreSet, NewDB)
    return nil, nil  // These return values are ignored.
}

Now `go generate` will create a new file with the generated code:

// File: wire_gen.go
// Code generated by Wire. DO NOT EDIT.
//go:generate wire
//+build !wireinject

func initUserStore(info ConnectionInfo) (*UserStore, error) {
    defaultConfig := NewDefaultConfig()
    db, err := NewDB(info)
    if err != nil {
        return nil, err
    }
    userStore, err := NewUserStore(defaultConfig, db)
    if err != nil {
        return nil, err
    }
    return userStore, nil
}

Any non-injector declarations are copied into the generated file. There is no dependency on Wire at runtime: all of the written code is just normal Go code.

As you can see, the output is very close to what a developer would write themselves. This was a trivial example with just three components, so writing the initializer by hand wouldn't be too painful, but Wire saves a lot of manual toil for components and applications with more complex dependency graphs.

How can I get involved and learn more?

The Wire README goes into more detail about how to use Wire and its more advanced features. There's also a tutorial that walks through using Wire in a simple application.

We appreciate any input you have about your experience with Wire! Wire's development is conducted on GitHub, so you can file an issue to tell us what could be better. For updates and discussion about the project, join the Go Cloud mailing list.

Thank you for taking the time to learn about Go Cloud's Wire. We’re excited to work with you to make Go the language of choice for developers building portable cloud applications.

By Robert van Gent

See the index for more articles.