The Go Blog

Go, the Go Community, and the Pandemic

Carmen Andoh, Russ Cox, and Steve Francia
25 March 2020

Go always comes second to more basic concerns like personal and family health and safety. Around the world, the past couple months have been terrible, and we are still at the start of this awful pandemic. There are days when it seems like working on anything related to Go should be considered a serious priority inversion.

But after we’ve done all we can to prepare ourselves and our families for whatever is coming, getting back to some approximation of a familiar routine and normal work is a helpful coping mechanism. In that spirit, we intend to keep working on Go and trying to help the Go community as much as we can.

In this post we want to share a few important notes about how the pandemic is affecting the Go community, a few things we’re doing to help, what you can do to help, and our plans for Go itself.

Conferences and Meetups

The Go community thrives on in-person conferences and meetups. We had anticipated 35 conferences this year and thousands of meetups, nearly all of which have now changed, been postponed, or been cancelled. We’ll keep the conferences wiki page updated as plans change.

We want to do everything we can to help support impacted Go conferences. We also want to support efforts to explore new ways for gophers to connect in the time of social distancing. In addition to honoring Google’s existing sponsorships, we are interested to offer support to people planning virtual conference alternatives through the rest of the year. If you are organizing a Go conference and have been impacted, or if you are considering holding a virtual alternative, please reach out to Carmen Andoh at candoh@google.com.

For conference organizers, the Gophers slack #conf-organizers channel is a place to discuss contingency plans, best practices, cancellation, and postponement support. It’s also a place to share idea for virtual events, to continue to connect and support the Go community.

For meetup organizers, the Go Developer Network can provide Zoom for Education licensing to meetups that want to start holding virtual meetings. If you host a meetup, or you’d like to, we encourage you to use this opportunity to get speakers from outside your region to present to your group. For more information, and to get involved, please join the Gophers slack #remotemeetup channel.

Online Training

The Go trainers you meet at conferences also travel the globe doing in-person training for companies that want help adopting Go. That in-person teaching is crucial to bringing new gophers into the community; we’re incredibly grateful to the trainers for the work they do. Unfortunately, on-site training contracts have all been cancelled for the next few months, and the trainers in our community have lost their primary (or sole) source of income. We encourage companies to consider virtual training and workshops during this difficult time. Most trainers are being flexible with pricing, scheduling, and class structure.

Job Postings

We know that the current downturn means that some gophers are looking for new jobs. The Go community has built a number of Go-specific job-posting sites, including Golang Cafe, Golang Projects, and We Love Go. The Gophers slack also has many job-hunting channels: search for “job” in the channel list. We encourage employers with any new openings to post in as many appropriate places as possible.

FOSS Responders

We are proud that Go is part of the broader open-source ecosystem. FOSS Responders is one effort to help the open-source ecosystem deal with the impacts of the pandemic. If you want to do something to help affected open-source communities, they are coordinating efforts and also have links to other efforts. And if you know of other open-source communities that need help, let them know about FOSS Responders.

COVID-19 Open-Source Help Desk

The COVID-19 Open-Source Help Desk aims to help virologists, epidemiologists, and other domain experts find quick answers to any problems they are having with open-source scientific computing software, from experts in that software, so they can focus their time on what they know best. If you are a developer or a scientific computing expert willing to help by answering the posts of the domain experts, visit the site to learn how to help.

U.S. Digital Response

For our gophers in the United States, the U.S. Digital Response is working to connect qualified volunteers to state and local governments that need digital help during this crisis. Quoting the web page, “If you have relevant experience (healthcare, data, engineering & product development, general management, operations, supply chain/procurement and more), can work autonomously through ambiguity, and are ready to jump into a high-intensity environment,” see the site for how to volunteer.

Plans for Go

Here on the Go team at Google, we recognize that the world around us is changing rapidly and that plans beyond the next couple weeks are not much more than hopeful guesses. That said, right now we are working on what we think are the most important projects for 2020. Like all of you, we’re at reduced capacity, so the work continues slower than planned.

Our analysis of the Go 2019 user survey is almost complete, and we hope to post it soon.

At least for now, we intend to keep to our timeline for Go 1.15, with the understanding that it will probably have fewer new features and improvements than we originally planned. We continue to do code reviews, issue triage, and proposal review.

Gopls is the language-aware backend supporting most Go editors today, and we continue to work toward its 1.0 release.

The new Go package and module site pkg.go.dev keeps getting better. We’ve been working on usability improvements and new features to better help users find and evaluate Go packages. We’ve also expanded the set of recognized licenses and improved the license detector, with more improvements to come.

Our Gopher values are what ground us, now more than ever. We are working extra hard to be friendly, welcoming, patient, thoughtful, respectful, and charitable. We hope everyone in the Go community will try to do the same.

We’ll continue to use this blog to let you know about important news for the Go ecosystem. In those moments when you’ve taken care of the much more important things going on in your life, we hope you’ll check in and see what we’ve been up to.

Thank you, as always, for using Go and being part of the Go community. We wish you all the best in these difficult times.

A new Go API for Protocol Buffers

Joe Tsai, Damien Neil, and Herbie Ong
2 March 2020

Introduction

We are pleased to announce the release of a major revision of the Go API for protocol buffers, Google's language-neutral data interchange format.

Motivations for a new API

The first protocol buffer bindings for Go were announced by Rob Pike in March of 2010. Go 1 would not be released for another two years.

In the decade since that first release, the package has grown and developed along with Go. Its users' requirements have grown too.

Many people want to write programs that use reflection to examine protocol buffer messages. The reflect package provides a view of Go types and values, but omits information from the protocol buffer type system. For example, we might want to write a function that traverses a log entry and clears any field annotated as containing sensitive data. The annotations are not part of the Go type system.

Another common desire is to use data structures other than the ones generated by the protocol buffer compiler, such as a dynamic message type capable of representing messages whose type is not known at compile time.

We also observed that a frequent source of problems was that the proto.Message interface, which identifies values of generated message types, does very little to describe the behavior of those types. When users create types that implement that interface (often inadvertently by embedding a message in another struct) and pass values of those types to functions expecting a generated message value, programs crash or behave unpredictably.

All three of these problems have a common cause, and a common solution: The Message interface should fully specify the behavior of a message, and functions operating on Message values should freely accept any type that correctly implements the interface.

Since it is not possible to change the existing definition of the Message type while keeping the package API compatible, we decided that it was time to begin work on a new, incompatible major version of the protobuf module.

Today, we're pleased to release that new module. We hope you like it.

Reflection

Reflection is the flagship feature of the new implementation. Similar to how the reflect package provides a view of Go types and values, the google.golang.org/protobuf/reflect/protoreflect package provides a view of values according to the protocol buffer type system.

A complete description of the protoreflect package would run too long for this post, but let's look at how we might write the log-scrubbing function we mentioned previously.

First, we'll write a .proto file defining an extension of the google.protobuf.FieldOptions type so we can annotate fields as containing sensitive information or not.

syntax = "proto3";
import "google/protobuf/descriptor.proto";
package golang.example.policy;
extend google.protobuf.FieldOptions {
    bool non_sensitive = 50000;
}

We can use this option to mark certain fields as non-sensitive.

message MyMessage {
    string public_name = 1 [(golang.example.policy.non_sensitive) = true];
}

Next, we will write a Go function which accepts an arbitrary message value and removes all the sensitive fields.

// Redact clears every sensitive field in pb.
func Redact(pb proto.Message) {
   // ...
}

This function accepts a proto.Message, an interface type implemented by all generated message types. This type is an alias for one defined in the protoreflect package:

type ProtoMessage interface{
    ProtoReflect() Message
}

To avoid filling up the namespace of generated messages, the interface contains only a single method returning a protoreflect.Message, which provides access to the message contents.

(Why an alias? Because protoreflect.Message has a corresponding method returning the original proto.Message, and we need to avoid an import cycle between the two packages.)

The protoreflect.Message.Range method calls a function for every populated field in a message.

m := pb.ProtoReflect()
m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
    // ...
    return true
})

The range function is called with a protoreflect.FieldDescriptor describing the protocol buffer type of the field, and a protoreflect.Value containing the field value.

The protoreflect.FieldDescriptor.Options method returns the field options as a google.protobuf.FieldOptions message.

opts := fd.Options().(*descriptorpb.FieldOptions)

(Why the type assertion? Since the generated descriptorpb package depends on protoreflect, the protoreflect package can't return the concrete options type without causing an import cycle.)

We can then check the options to see the value of our extension boolean:

if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
    return true // don't redact non-sensitive fields
}

Note that we are looking at the field descriptor here, not the field value. The information we're interested in lies in the protocol buffer type system, not the Go one.

This is also an example of an area where we have simplified the proto package API. The original proto.GetExtension returned both a value and an error. The new proto.GetExtension returns just a value, returning the default value for the field if it is not present. Extension decoding errors are reported at Unmarshal time.

Once we have identified a field that needs redaction, clearing it is simple:

m.Clear(fd)

Putting all the above together, our complete redaction function is:

// Redact clears every sensitive field in pb.
func Redact(pb proto.Message) {
    m := pb.ProtoReflect()
    m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
        opts := fd.Options().(*descriptorpb.FieldOptions)
        if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
            return true
        }
        m.Clear(fd)
        return true
    })
}

A more complete implementation might recursively descend into message-valued fields. We hope that this simple example gives a taste of protocol buffer reflection and its uses.

Versions

We call the original version of Go protocol buffers APIv1, and the new one APIv2. Because APIv2 is not backwards compatible with APIv1, we need to use different module paths for each.

(These API versions are not the same as the versions of the protocol buffer language: proto1, proto2, and proto3. APIv1 and APIv2 are concrete implementations in Go that both support the proto2 and proto3 language versions.)

The github.com/golang/protobuf module is APIv1.

The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. (We considered google.golang.org/protobuf/v2, to make it clear that this is the second major version of the API, but settled on the shorter path as being the better choice in the long term.)

We know that not all users will move to a new major version of a package at the same rate. Some will switch quickly; others may remain on the old version indefinitely. Even within a single program, some parts may use one API while others use another. It is essential, therefore, that we continue to support programs that use APIv1.

  • github.com/golang/protobuf@v1.3.4 is the most recent pre-APIv2 version of APIv1.

  • github.com/golang/protobuf@v1.4.0 is a version of APIv1 implemented in terms of APIv2. The API is the same, but the underlying implementation is backed by the new one. This version contains functions to convert between the APIv1 and APIv2 proto.Message interfaces to ease the transition between the two.

  • google.golang.org/protobuf@v1.20.0 is APIv2. This module depends upon github.com/golang/protobuf@v1.4.0, so any program which uses APIv2 will automatically pick a version of APIv1 which integrates with it.

(Why start at version v1.20.0? To provide clarity. We do not anticipate APIv1 to ever reach v1.20.0, so the version number alone should be enough to unambiguously differentiate between APIv1 and APIv2.)

We intend to maintain support for APIv1 indefinitely.

This organization ensures that any given program will use only a single protocol buffer implementation, regardless of which API version it uses. It permits programs to adopt the new API gradually, or not at all, while still gaining the advantages of the new implementation. The principle of minimum version selection means that programs may remain on the old implementation until the maintainers choose to update to the new one (either directly, or by updating a dependency).

Additional features of note

The google.golang.org/protobuf/encoding/protojson package converts protocol buffer messages to and from JSON using the canonical JSON mapping, and fixes a number of issues with the old jsonpb package that were difficult to change without causing problems for existing users.

The google.golang.org/protobuf/types/dynamicpb package provides an implementation of proto.Message for messages whose protocol buffer type is derived at runtime.

The google.golang.org/protobuf/testing/protocmp package provides functions to compare protocol buffer messages with the github.com/google/cmp package.

The google.golang.org/protobuf/compiler/protogen package provides support for writing protocol compiler plugins.

Conclusion

The google.golang.org/protobuf module is a major overhaul of Go's support for protocol buffers, providing first-class support for reflection, custom message implementations, and a cleaned up API surface. We intend to maintain the previous API indefinitely as a wrapper of the new one, allowing users to adopt the new API incrementally at their own pace.

Our goal in this update is to improve upon the benefits of the old API while addressing its shortcomings. As we completed each component of the new implementation, we put it into use within Google's codebase. This incremental rollout has given us confidence in both the usability of the new API and the performance and correctness of the new implementation. We believe it is production ready.

We are excited about this release and hope that it will serve the Go ecosystem for the next ten years and beyond!

Go 1.14 is released

Alex Rakoczy
25 February 2020

Today the Go team is very happy to announce the release of Go 1.14. You can get it from the download page.

Some of the highlights include:

For the complete list of changes and more information about the improvements above, see the Go 1.14 release notes.

We want to thank everyone who contributed to this release by writing code, filing bugs, providing feedback, and/or testing the beta and release candidate. Your contributions and diligence helped to ensure that Go 1.14 is as stable as possible. That said, if you notice any problems, please file an issue.

We hope you enjoy the new release!

Next steps for pkg.go.dev

Julie Qiu
31 January 2020

Introduction

In 2019, we launched go.dev, a new hub for Go developers.

As part of the site, we also launched pkg.go.dev, a central source of information about Go packages and modules. Like godoc.org, pkg.go.dev serves Go documentation. However, it also understands modules and has information about past versions of a package!

Throughout this year, we will be adding features to pkg.go.dev to help our users better understand their dependencies and help them make better decisions around what libraries to import.

Redirecting godoc.org requests to pkg.go.dev

To minimize confusion about which site to use, later this year we are planning to redirect traffic from godoc.org to the corresponding page on pkg.go.dev. We need your help to ensure that pkg.go.dev addresses all of our users' needs. We encourage everyone to begin using pkg.go.dev today for all of their needs and provide feedback.

Your feedback will inform our transition plan, with the goal of making pkg.go.dev our primary source of information and documentation for packages and modules. We’re sure there are things that you want to see on pkg.go.dev, and we want to hear from you about what those features are!

You can share your feedback with us on these channels:

As part of this transition, we will also be discussing plans for API access to pkg.go.dev. We will be posting updates on Go issue 33654.

Frequently asked questions

Since our launch in November, we’ve received tons of great feedback about pkg.go.dev from Go users. For the remainder of this post, we thought it would be helpful to answer some frequently asked questions.

My package doesn’t show up on pkg.go.dev! How do I add it?

We monitor the Go Module Index regularly for new packages to add to pkg.go.dev. If you don’t see a package on pkg.go.dev, you can add it by fetching the module version from proxy.golang.org. See go.dev/about for instructions.

My package has license restrictions. What’s wrong with it?

We understand it can be a frustrating experience to not be able to see the package you want in its entirety on pkg.go.dev. We appreciate your patience as we improve our license detection algorithm.

Since our launch in November, we've made the following improvements:

  • Updated our license policy to include the list of licenses that we detect and recognize
  • Worked with the licensecheck team to improve detection for copyright notices
  • Established a manual review process for special cases

As always, our license policy is at pkg.go.dev/license-policy. If you are having issues, feel free to file an issue on the Go issue tracker, or email go-discovery-feedback@google.com so that we can work with you directly!

Will pkg.go.dev be open-sourced so I can run it at work for my private code?

We understand that corporations with private code want to run a documentation server that provides module support. We want to help meet that need, but we feel we don’t yet understand it as well as we need to.

We’ve heard from users that running the godoc.org server is more complex than it should be, because it is designed for serving at public internet scale instead of just within a company. We believe the current pkg.go.dev server would have the same problem.

We think a new server is more likely to be the right answer for use with private code, instead of exposing every company to the complexity of running the internet-scale pkg.go.dev codebase. In addition to serving documentation, a new server could also serve information to goimports and gopls.

If you want to run such a server, please fill out this 3-5 minute survey to help us better understand your needs. This survey will be available until March 1st, 2020.

We’re excited about the future of pkg.go.dev in 2020, and we hope you are too! We look forward to hearing your feedback and working with the Go community on this transition.

Proposals for Go 1.15

Robert Griesemer, for the Go team
28 January 2020

Status

We are close to the Go 1.14 release, planned for February assuming all goes well, with an RC1 candidate almost ready. Per the process outlined in the Go 2, here we come! blog post, it is again the time in our development and release cycle to consider if and what language or library changes we might want to include for our next release, Go 1.15, scheduled for August of this year.

The primary goals for Go remain package and version management, better error handling support, and generics. Module support is in good shape and getting better with each day, and we are also making progress on the generics front (more on that later this year). Our attempt seven months ago at providing a better error handling mechanism, the try proposal, met good support but also strong opposition and we decided to abandon it. In its aftermath there were many follow-up proposals, but none of them seemed convincing enough, clearly superior to the try proposal, or less likely to cause similar controversy. Thus, we have not further pursued changes to error handling for now. Perhaps some future insight will help us to improve upon the status quo.

Proposals

Given that modules and generics are actively being worked on, and with error handling changes out of the way for the time being, what other changes should we pursue, if any? There are some perennial favorites such as requests for enums and immutable types, but none of those ideas are sufficiently developed yet, nor are they urgent enough to warrant a lot of attention by the Go team, especially when also considering the cost of making a language change.

After reviewing all potentially viable proposals, and more importantly, because we don’t want to incrementally add new features without a long-term plan, we concluded that it is better to hold off with major changes this time. Instead we concentrate on a couple of new vet checks and a minor adjustment to the language. We have selected the following three proposals:

#32479. Diagnose string(int) conversion in go vet.

We were planning to get this done for the upcoming Go 1.14 release but we didn’t get around to it, so here it is again. The string(int) conversion was introduced early in Go for convenience, but it is confusing to newcomers (string(10) is "\n" not "10") and not justified anymore now that the conversion is available in the unicode/utf8 package. Since removing this conversion is not a backwards-compatible change, we propose to start with a vet error instead.

#4483. Diagnose impossible interface-interface type assertions in go vet.

Currently, Go permits any type assertion x.(T) (and corresponding type switch case) where the type of x and T are interfaces. Yet, if both x and T have a method with the same name but different signatures it is impossible for any value assigned to x to also implement T; such type assertions will always fail at runtime (panic or evaluate to false). Since we know this at compile time, the compiler might as well report an error. Reporting a compiler error in this case is not a backwards-compatible change, thus we also propose to start with a vet error instead.

#28591. Constant-evaluate index and slice expressions with constant strings and indices.

Currently, indexing or slicing a constant string with a constant index, or indices, produces a non-constant byte or string value, respectively. But if all operands are constant, the compiler can constant-evaluate such expressions and produce a constant (possibly untyped) result. This is a fully backward-compatible change and we propose to make the necessary adjustments to the spec and compilers.

(Correction: We found out after posting that this change is not backward-compatible; see comment for details.)

Timeline

We believe that none of these three proposals are controversial but there’s always a chance that we missed something important. For that reason we plan to have the proposals implemented at the beginning of the Go 1.15 release cycle (at or shortly after the Go 1.14 release) so that there is plenty of time to gather experience and provide feedback. Per the proposal evaluation process, the final decision will be made at the end of the development cycle, at the beginning of May, 2020.

And one more thing...

We receive many more language change proposals (issues labeled LanguageChange) than we can review thoroughly. For instance, just for error handling alone, there are 57 issues, of which five are currently still open. Since the cost of making a language change, no matter how small, is high and the benefits are often unclear, we must err on the side of caution. Consequently, most language change proposals get rejected sooner or later, sometimes with minimal feedback. This is unsatisfactory for all parties involved. If you have spent a lot of time and effort outlining your idea in detail, it would be nice to not have it immediately rejected. On the flip side, because the general proposal process is deliberately simple, it is very easy to create language change proposals that are only marginally explored, causing the review committee significant amounts of work. To improve this experience for everybody we are adding a new questionnaire for language changes: filling out that template will help reviewers evaluate proposals more efficiently because they don’t need to try to answer those questions themselves. And hopefully it will also provide better guidance for proposers by setting expectations right from the start. This is an experiment that we will refine over time as needed.

Thank you for helping us improve the Go experience!

See the index for more articles.