The Go Blog

Go Developer Survey 2019 Results

Todd Kulesza
20 April 2020

What a response!

I want to start with an enormous thank you to the thousands of Go developers who participated in this year’s survey. For 2019, we saw 10,975 responses, nearly twice as many as last year! On behalf of the rest of the team, I cannot adequately stress how much we appreciate you taking the time and effort to tell us about your experiences with Go. Thank you!

A note about prior years

Sharp-eyed readers may notice that our year-over-year comparisons don’t quite square with numbers we’ve shared in the past. The reason is that from 2016–2018, we calculated percentages for each question using the total number of people who started the survey as the denominator. While that’s nice and consistent, it ignores the fact that not everyone finishes the survey—up to 40% of participants stop before reaching the final page, which meant questions that occurred later in the survey appeared to perform worse solely because they were later. Thus, this year we’ve recalculated all of our results (including the 2016–2018 responses shown in this post) to use the number of people who responded to a given question as the denominator for that question. We’ve included the number of 2019 responses for each chart—in the form of "n=[number of respondents]" on the x-axis or in the chart’s legend—to give readers a better understanding of the weight of evidence underlying each finding.

Similarly, we learned that in prior surveys options that appeared earlier in response lists had a disproportionate response rate. To address this, we added an element of randomization into the survey. Some of our multiple-choice questions have lists of choices with no logical ordering, such as "I write the following in Go: [list of types of applications]". Previously these choices had been alphabetized, but for 2019 they were presented in a random order to each participant. This means year-over-year comparison for certain questions are invalid for 2018 → 2019, but trends from 2016–2018 are not invalidated. You can think of this as setting a more accurate baseline for 2019. We retained alphabetical ordering in cases where respondents are likely to scan for a particular name, such as their preferred editor. We explicitly call out which questions this applies to below.

A third major change was to improve our analysis of questions with open-ended, free-text responses. Last year we used machine learning to roughly—but quickly—categorize these responses. This year two researchers manually analyzed and categorized these responses, allowing for a more granular analysis but preventing valid comparisons with last year’s numbers. Like the randomization discussed above, the purpose of this change is to give us a reliable baseline for 2019 onward.

Without further ado…

This is a long post. Here’s the tl;dr of our major findings:

  • The demographics of our respondents are similar to Stack Overflow’s survey respondents, which increases our confidence that these results are representative of the larger Go developer audience.
  • A majority of respondents use Go every day, and this number has been trending up each year.
  • Go’s use is still concentrated in technology companies, but Go is increasingly found in a wider variety of industries, such as finance and media.
  • Methodology changes showed us that most of our year-over-year metrics are stable and higher than we previously realized.
  • Respondents are using Go to solve similar problems, particularly building API/RPC services and CLIs, regardless of the size of organization they work at.
  • Most teams try to update to the latest Go release quickly; when third-party providers are late to support the current Go release, this creates an adoption blocker for developers.
  • Almost everyone in the Go ecosystem is now using modules, but some confusion around package management remains.
  • High-priority areas for improvement include improving the developer experience for debugging, working with modules, and working with cloud services.
  • VS Code and GoLand have continued to see increased use; they’re now preferred by 3 out of 4 respondents.

Who did we hear from?

This year we asked some new demographic questions to help us better understand the people who’ve responded to this survey. In particular, we asked about the duration of professional programming experience and the size of the organizations where people work. These were modeled on questions that StackOverflow asks in their annual survey, and the distribution of responses we saw is very close to StackOverflow’s 2019 results. Our take-away is the respondents to this survey have similar levels of professional experience and proportional representation of different sizes of organizations as the StackOverflow survey audience (with the obvious difference that we’re primarily hearing from developers working with Go). That increases our confidence when generalizing these findings to the estimated 1 million Go developers worldwide. These demographic questions will also help us in the future to identify which year-over-year changes may be the result of a shift in who responded to the survey, rather than changes in sentiment or behavior.

Looking at Go experience, we see that a majority of respondents (56%) are relatively new to Go, having used it for less than two years. Majorities also said they use Go at work (72%) and outside of work (62%). The percentage of respondents using Go professionally appears to be trending up each year.

As you can see in the chart below, in 2018 we saw a spike in these numbers, but that increase disappeared this year. This is one of many signals suggesting that the audience who answered the survey in 2018 was significantly different than in the other three years. In this case they were significantly more likely to be using Go outside of work and a different language while at work, but we see similar outliers across multiple survey questions.

Respondents who have been using Go the longest have different backgrounds than newer Go developers. These Go veterans were more likely to claim expertise in C/C++ and less likely to claim expertise in JavaScript, TypeScript, and PHP. One caveat is that this is self-reported "expertise"; it may be more helpful to think of it instead as "familiarity". Python appears to be the language (other than Go) familiar to the most respondents, regardless of how long they’ve been working with Go.

Last year we asked about which industries respondents work in, finding that a majority reported working in software, internet, or web service companies. This year it appears respondents represent a broader range of industries. However, we also simplified the list of industries to reduce confusion from potentially overlapping categories (e.g., the separate categories for "Software" and "Internet / web services" from 2018 were combined into "Technology" for 2019). Thus, this isn’t strictly an apples-to-apples comparison. For example, it’s possible that one effect of simplifying the category list was to reduce the use of the "Software" category as a catch-all for respondents writing Go software for an industry that wasn’t explicitly listed.

Go is a successful open-source project, but that doesn’t mean the developers working with it are also writing free or open-source software. As in prior years, we found that most respondents are not frequent contributors to Go open-source projects, with 75% saying they do so "infrequently" or "never". As the Go community expands, we see the proportion of respondents who’ve never contributed to Go open-source projects slowly trending up.

Developer tools

As in prior years, the vast majority of survey respondents reported working with Go on Linux and macOS systems. This is one area of strong divergence between our respondents and StackOverflow’s 2019 results: in our survey, only 20% of respondents use Windows as a primary development platform, while for StackOverflow it was 45% of respondents. Linux is used by 66% and macOS by 53%—both much higher than the StackOverflow audience, which reported 25% and 30%, respectively.

The trend in editor consolidation has continued this year. GoLand saw the sharpest increase in use this year, rising from 24% → 34%. VS Code’s growth slowed, but it remains the most popular editor among respondents at 41%. Combined, these two editors are now preferred by 3 out of 4 respondents.

Every other editor saw a small decrease. This doesn’t mean those editors aren’t being used at all, but they’re not what respondents say they prefer to use for writing Go code.

This year we added a question about internal Go documentation tooling, such as gddo. A small minority of respondents (6%) reported that their organization runs its own Go documentation server, though this proportion nearly doubles (to 11%) when we look at respondents at large organizations (those with at least 5,000 employees). A follow-up asked of respondents who said their organization had stopped running its own documentation server suggests that the top reason to retire their server was a combination of low perceived benefits (23%) versus the amount of effort required to initially set it up and maintain it (38%).

Sentiments towards Go

Large majorities of respondents agreed that Go is working well for their teams (86%) and that they’d prefer to use it for their next project (89%). We also found that over half of respondents (59%) believe Go is critical to the success of their companies. All of these metrics have remained stable since 2016.

Normalizing the results changed most of these numbers for prior years. For example, the percentage of respondents who agreed with the statement "Go is working well for my team" was previously in the 50’s and 60’s because of participant drop-off; when we remove participants who never saw the question, we see it’s been fairly stable since 2016.

Looking at sentiments toward problem solving in the Go ecosystem, we see similar results. Large percentages of respondents agreed with each statement (82%–88%), and these rates have been largely stable over the past four years.

This year we took a more nuanced look at satisfaction across industries to establish a baseline. Overall, respondents were positive about using Go at work, regardless of industry sector. We do see small variations in dissatisfaction in a few areas, most notably manufacturing, which we plan to investigate with follow-up research. Similarly, we asked about satisfaction with—and the importance of—various aspects of Go development. Pairing these measures together highlighted three topics of particular focus: debugging (including debugging concurrency), using modules, and using cloud services. Each of these topics was rated "very" or "critically" important by a majority of respondents but had significantly lower satisfaction scores compared to other topics.

Turning to sentiments toward the Go community, we see some differences from prior years. First, there is a dip in the percentage of respondents who agreed with the statement "I feel welcome in the Go community", from 82% to 75%. Digging deeper revealed that the proportion of respondents who "slightly" or "moderately agreed" decreased, while the proportions who "neither agree nor disagree" and "strongly agree" both increased (up 5 and 7 points, respectively). This polarizing split suggests two or more groups whose experiences in the Go community are diverging, and is thus another area we plan to further investigate.

The other big differences are a clear upward trend in responses to the statement "I feel welcome to contribute to the Go project" and a large year-over-year increase in the proportion of respondents who feel Go’s project leadership understands their needs.

All of these results show a pattern of higher agreement correlated with increased Go experience, beginning at about two years. In other words, the longer a respondent has been using Go, the more likely they were to agree with each of these statements.

This likely comes as no surprise, but people who responded to the Go Developer Survey tended to like Go. However, we also wanted to understand which other languages respondents enjoy working with. Most of these numbers have not significantly changed from prior years, with two exceptions: TypeScript (which has increased 10 points), and Rust (up 7 points). When we break these results down by duration of Go experience, we see the same pattern as we found for language expertise. In particular, Python is the language and ecosystem that Go developers are most likely to also enjoy building with.

In 2018 we first asked the "Would you recommend…" Net Promoter Score (NPS) question, yielding a score of 61. This year our NPS result is a statistically unchanged 60 (67% "promoters" minus 7% "detractors").

Working with Go

Building API/RPC services (71%) and CLIs (62%) remain the most common uses of Go. The chart below appears to show major changes from 2018, but these are most likely the result of randomizing the order of choices, which used to be listed alphabetically: 3 of the 4 choices beginning with ’A’ decreased, while everything else remained stable or increased. Thus, this chart is best interpreted as a more accurate baseline for 2019 with trends from 2016–2018. For example, we believe that the proportion of respondents building web services which return HTML has been decreasing since 2016 but were likely undercounted because this response was always at the bottom of a long list of choices. We also broke this out by organization size and industry but found no significant differences: it appears respondents use Go in roughly similar ways whether they work at a small tech start-up or a large retail enterprise.

A related question asked about the larger areas in which respondents work with Go. The most common area by far was web development (66%), but other common areas included databases (45%), network programming (42%), systems programming (38%), and DevOps tasks (37%).

In addition to what respondents are building, we also asked about some of the development techniques they use. A large majority of respondents said they depend upon text logs for debugging (88%), and their free-text responses suggest this is because alternative tooling is challenging to use effectively. However, local stepwise debugging (e.g., with Delve), profiling, and testing with the race detector were not uncommon, with ~50% of respondents depending upon at least one of these techniques.

Regarding package management, we found that the vast majority of respondents have adopted modules for Go (89%). This has been a big shift for developers, and nearly the entire community appears to be going through it simultaneously.

We also found that 75% of respondents evaluate the current Go release for production use, with an additional 12% waiting one release cycle. This suggests a large majority of Go developers are using (or at the least, trying to use) the current or previous stable release, highlighting the importance for platform-as-a-service providers to quickly support new stable releases of Go.

Go in the clouds

Go was designed with modern distributed computing in mind, and we want to continue to improve the developer experience of building cloud services with Go. This year we expanded the questions we asked about cloud development to better understand how respondents are working with cloud providers, what they like about the current developer experience, and what can be improved. As mentioned earlier, some of the 2018 results appear to be outliers, such as an unexpectedly low result for self-owned servers, and an unexpectedly high result for GCP deployments.

We see two clear trends:

  1. The three largest global cloud providers (Amazon Web Services, Google Cloud Platform, and Microsoft Azure) all appear to be trending up in usage among survey respondents, while most other providers are used by a smaller proportion of respondents each year.
  2. On-prem deployments to self-owned or company-owned servers continue to decrease and are now statistically tied with AWS (44% vs. 42%) as the most common deployment targets.

Looking at which types of cloud platforms respondents are using, we see differences between the major providers. Respondents deploying to AWS and Azure were most likely to be using VMs directly (65% and 51%, respectively), while those deploying to GCP were almost twice as likely to be using the managed Kubernetes platform (GKE, 64%) than VMs (35%). We also found that respondents deploying to AWS were equally likely to be using a managed Kubernetes platform (32%) as they were to be using a managed serverless platform (AWS Lambda, 33%). Both GCP (17%) and Azure (7%) had lower proportions of respondents using serverless platforms, and free-text responses suggest a primary reason was delayed support for the latest Go runtime on these platforms.

Overall, a majority of respondents were satisfied with using Go on all three major cloud providers. Respondents reported similar satisfaction levels with Go development for AWS (80% satisfied) and GCP (78%). Azure received a lower satisfaction score (57% satisfied), and free-text responses suggest that the main driver was a perception that Go lacks first-class support on this platform (25% of free-text responses). Here, "first-class support" refers to always staying up-to-date with the latest Go release, and ensuring new features are available to Go developers at time of launch. This was the same top pain-point reported by respondents using GCP (14%), and particularly focused on support for the latest Go runtime in serverless deployments. Respondents deploying to AWS, in contrast, were most likely to say the SDK could use improvements, such as being more idiomatic (21%). SDK improvements were also the second most common request for both GCP (9%) and Azure (18%) developers.

Pain points

The top reasons respondents say they are unable to use Go more remain working on a project in another language (56%), working on a team that prefers to use another language (37%), and the lack of a critical feature in Go itself (25%).

This was one of the questions where we randomized the choice list, so year-over-year comparisons aren’t valid, though 2016–2018 trends are. For example, we are confident that the number of developers unable to use Go more frequently because their team prefers a different language is decreasing each year, but we don’t know whether that decrease dramatically accelerated this year, or was always a bit lower than our 2016–2018 numbers estimated.

The top two adoption blockers (working on an existing non-Go project and working on a team that prefers a different language) don’t have direct technical solutions, but the remaining blockers might. Thus, this year we asked for more details, to better understand how we might help developers increase their use of Go. The charts in the remainder of this section are based on free-text responses which were manually categorized, so they have very long tails; categories totalling less than 3% of the total responses have been grouped into the "Other" category for each chart. A single response may mention multiple topics, thus charts do not not sum to 100%.

Among the 25% of respondents who said Go lacks language features they need, 79% pointed to generics as a critical missing feature. Continued improvements to error handling (in addition to the Go 1.13 changes) was cited by 22%, while 13% requested more functional programming features, particularly built-in map/filter/reduce functionality. To be clear, these numbers are from the subset of respondents who said they would be able to use Go more were it not missing one or more critical features they need, not the entire population of survey respondents.

Respondents who said Go "isn’t an appropriate language" for what they work on had a wide variety of reasons and use-cases. The most common was that they work on some form of front-end development (22%), such as GUIs for web, desktop, or mobile. Another common response was that the respondent said they worked in a domain with an already-dominant language (9%), making it a challenge to use something different. Some respondents also told us which domain they were referring to (or simply mentioned a domain without mentioning another language being more common), which we show via the "I work on [domain]" rows below. An additional top reason cited by respondents was a need for better performance (9%), particularly for real-time computing.

The biggest challenges respondents reported remain largely consistent with last year. Go’s lack of generics and modules/package management still top the list (15% and 12% of responses, respectively), and the proportion of respondents highlighting tooling problems increased. These numbers are different from the above charts because this question was asked of all respondents, regardless of what they said their biggest Go adoption blockers were. All three of these are areas of focus for the Go team this year, and we hope to greatly improve the developer experience, particularly around modules, tooling, and the getting started experience, in the coming months.

Diagnosing faults and performance issues can be challenging in any language. Respondents told us their top challenge for both of these was not something specific to Go’s implementation or tooling, but a more fundamental issue: a self-reported lack of knowledge, experience, or best practices. We hope to help address these knowledge gaps via documentation and other educational materials later this year. The other major problems do involve tooling, specifically a perceived unfavorable cost/benefit trade-off to learning/using Go’s debugging and profiling tooling, and challenges making the tooling work in various environments (e.g., debugging in containers, or getting performance profiles from production systems).

Finally, when we asked what would most improve Go support in respondents’ editing environment, the most common response was for general improvements or better support for the language server (gopls, 19%). This was expected, as gopls replaces about 80 extant tools and is still in beta. When respondents were more specific about what they’d like to see improved, they were most likely to report the debugging experience (14%) and faster or more reliable code completion (13%). A number of participants also explicitly referenced the need to frequently restart VS Code when using gopls (8%); in the time since this survey was in the field (late November – early December 2019), many of these gopls improvements have already landed, and this continues to be a high-priority area for the team.

The Go community

Roughly two thirds of respondents used Stack Overflow to answer their Go-related questions (64%). The other top sources of answers were godoc.org (47%), directly reading source code (42%), and golang.org (33%).

The long tail on the previous chart highlights the large variety of different sources (nearly all of them community-driven) and modalities that respondents rely on to overcome challenges while developing with Go. Indeed, for many Gophers, this may be one of their main points of interaction with the larger community: as our community expands, we’ve seen higher and higher proportions of respondents who do not attend any Go-related events. For 2019, that proportion nearly reached two thirds of respondents (62%).

Due to updated Google-wide privacy guidelines, we can no longer ask about which countries respondents live in. Instead we asked about preferred spoken/written language as a very rough proxy for Go’s worldwide usage, with the benefit of providing data for potential localization efforts.

Because this survey is in English, there is likely a strong bias toward English speakers and people from areas where English is a common second or third language. Thus, the non-English numbers should be interpreted as likely minimums rather than an approximation of Go’s global audience.

We found 12% of respondents identify with a traditionally underrepresented group (e.g., ethnicity, gender identity, et al.) and 3% identify as female. (This question should have said "woman" instead of "female". The mistake has been corrected in our draft survey for 2020, and we apologize for it.) We strongly suspect this 3% is undercounting women in the Go community. For example, we know women software developers in the US respond to the StackOverflow Developer Survey at about half the rate we’d expect based on US employment figures (11% vs 20%). Since we don’t know the proportion of responses in the US, we can’t safely extrapolate from these numbers beyond saying the actual proportion is likely higher than 3%. Furthermore, GDPR required us to change how we ask about sensitive information, which includes gender and traditionally underrepresented groups. Unfortunately these changes prevent us from being able to make valid comparisons of these numbers with prior years.

Respondents who identified with underrepresented groups or preferred not to answer this question showed higher rates of disagreement with the statement "I feel welcome in the Go community" (8% vs. 4%) than those who do not identify with an underrepresented group, highlighting the importance of our continued outreach efforts.

Conclusion

We hope you’ve enjoyed seeing the results of our 2019 developer survey. Understanding developers’ experiences and challenges helps us plan and prioritize work for 2020. Once again, an enormous thank you to everyone who contributed to this survey—your feedback is helping to steer Go’s direction in the coming year and beyond.

Go, the Go Community, and the Pandemic

Carmen Andoh, Russ Cox, and Steve Francia
25 March 2020

Go always comes second to more basic concerns like personal and family health and safety. Around the world, the past couple months have been terrible, and we are still at the start of this awful pandemic. There are days when it seems like working on anything related to Go should be considered a serious priority inversion.

But after we’ve done all we can to prepare ourselves and our families for whatever is coming, getting back to some approximation of a familiar routine and normal work is a helpful coping mechanism. In that spirit, we intend to keep working on Go and trying to help the Go community as much as we can.

In this post we want to share a few important notes about how the pandemic is affecting the Go community, a few things we’re doing to help, what you can do to help, and our plans for Go itself.

Conferences and Meetups

The Go community thrives on in-person conferences and meetups. We had anticipated 35 conferences this year and thousands of meetups, nearly all of which have now changed, been postponed, or been cancelled. We’ll keep the conferences wiki page updated as plans change.

We want to do everything we can to help support impacted Go conferences. We also want to support efforts to explore new ways for gophers to connect in the time of social distancing. In addition to honoring Google’s existing sponsorships, we are interested to offer support to people planning virtual conference alternatives through the rest of the year. If you are organizing a Go conference and have been impacted, or if you are considering holding a virtual alternative, please reach out to Carmen Andoh at candoh@google.com.

For conference organizers, the Gophers slack #conf-organizers channel is a place to discuss contingency plans, best practices, cancellation, and postponement support. It’s also a place to share idea for virtual events, to continue to connect and support the Go community.

For meetup organizers, the Go Developer Network can provide Zoom for Education licensing to meetups that want to start holding virtual meetings. If you host a meetup, or you’d like to, we encourage you to use this opportunity to get speakers from outside your region to present to your group. For more information, and to get involved, please join the Gophers slack #remotemeetup channel.

Online Training

The Go trainers you meet at conferences also travel the globe doing in-person training for companies that want help adopting Go. That in-person teaching is crucial to bringing new gophers into the community; we’re incredibly grateful to the trainers for the work they do. Unfortunately, on-site training contracts have all been cancelled for the next few months, and the trainers in our community have lost their primary (or sole) source of income. We encourage companies to consider virtual training and workshops during this difficult time. Most trainers are being flexible with pricing, scheduling, and class structure.

Job Postings

We know that the current downturn means that some gophers are looking for new jobs. The Go community has built a number of Go-specific job-posting sites, including Golang Cafe, Golang Projects, and We Love Go. The Gophers slack also has many job-hunting channels: search for “job” in the channel list. We encourage employers with any new openings to post in as many appropriate places as possible.

FOSS Responders

We are proud that Go is part of the broader open-source ecosystem. FOSS Responders is one effort to help the open-source ecosystem deal with the impacts of the pandemic. If you want to do something to help affected open-source communities, they are coordinating efforts and also have links to other efforts. And if you know of other open-source communities that need help, let them know about FOSS Responders.

COVID-19 Open-Source Help Desk

The COVID-19 Open-Source Help Desk aims to help virologists, epidemiologists, and other domain experts find quick answers to any problems they are having with open-source scientific computing software, from experts in that software, so they can focus their time on what they know best. If you are a developer or a scientific computing expert willing to help by answering the posts of the domain experts, visit the site to learn how to help.

U.S. Digital Response

For our gophers in the United States, the U.S. Digital Response is working to connect qualified volunteers to state and local governments that need digital help during this crisis. Quoting the web page, “If you have relevant experience (healthcare, data, engineering & product development, general management, operations, supply chain/procurement and more), can work autonomously through ambiguity, and are ready to jump into a high-intensity environment,” see the site for how to volunteer.

Plans for Go

Here on the Go team at Google, we recognize that the world around us is changing rapidly and that plans beyond the next couple weeks are not much more than hopeful guesses. That said, right now we are working on what we think are the most important projects for 2020. Like all of you, we’re at reduced capacity, so the work continues slower than planned.

Our analysis of the Go 2019 user survey is almost complete, and we hope to post it soon.

At least for now, we intend to keep to our timeline for Go 1.15, with the understanding that it will probably have fewer new features and improvements than we originally planned. We continue to do code reviews, issue triage, and proposal review.

Gopls is the language-aware backend supporting most Go editors today, and we continue to work toward its 1.0 release.

The new Go package and module site pkg.go.dev keeps getting better. We’ve been working on usability improvements and new features to better help users find and evaluate Go packages. We’ve also expanded the set of recognized licenses and improved the license detector, with more improvements to come.

Our Gopher values are what ground us, now more than ever. We are working extra hard to be friendly, welcoming, patient, thoughtful, respectful, and charitable. We hope everyone in the Go community will try to do the same.

We’ll continue to use this blog to let you know about important news for the Go ecosystem. In those moments when you’ve taken care of the much more important things going on in your life, we hope you’ll check in and see what we’ve been up to.

Thank you, as always, for using Go and being part of the Go community. We wish you all the best in these difficult times.

A new Go API for Protocol Buffers

Joe Tsai, Damien Neil, and Herbie Ong
2 March 2020

Introduction

We are pleased to announce the release of a major revision of the Go API for protocol buffers, Google's language-neutral data interchange format.

Motivations for a new API

The first protocol buffer bindings for Go were announced by Rob Pike in March of 2010. Go 1 would not be released for another two years.

In the decade since that first release, the package has grown and developed along with Go. Its users' requirements have grown too.

Many people want to write programs that use reflection to examine protocol buffer messages. The reflect package provides a view of Go types and values, but omits information from the protocol buffer type system. For example, we might want to write a function that traverses a log entry and clears any field annotated as containing sensitive data. The annotations are not part of the Go type system.

Another common desire is to use data structures other than the ones generated by the protocol buffer compiler, such as a dynamic message type capable of representing messages whose type is not known at compile time.

We also observed that a frequent source of problems was that the proto.Message interface, which identifies values of generated message types, does very little to describe the behavior of those types. When users create types that implement that interface (often inadvertently by embedding a message in another struct) and pass values of those types to functions expecting a generated message value, programs crash or behave unpredictably.

All three of these problems have a common cause, and a common solution: The Message interface should fully specify the behavior of a message, and functions operating on Message values should freely accept any type that correctly implements the interface.

Since it is not possible to change the existing definition of the Message type while keeping the package API compatible, we decided that it was time to begin work on a new, incompatible major version of the protobuf module.

Today, we're pleased to release that new module. We hope you like it.

Reflection

Reflection is the flagship feature of the new implementation. Similar to how the reflect package provides a view of Go types and values, the google.golang.org/protobuf/reflect/protoreflect package provides a view of values according to the protocol buffer type system.

A complete description of the protoreflect package would run too long for this post, but let's look at how we might write the log-scrubbing function we mentioned previously.

First, we'll write a .proto file defining an extension of the google.protobuf.FieldOptions type so we can annotate fields as containing sensitive information or not.

syntax = "proto3";
import "google/protobuf/descriptor.proto";
package golang.example.policy;
extend google.protobuf.FieldOptions {
    bool non_sensitive = 50000;
}

We can use this option to mark certain fields as non-sensitive.

message MyMessage {
    string public_name = 1 [(golang.example.policy.non_sensitive) = true];
}

Next, we will write a Go function which accepts an arbitrary message value and removes all the sensitive fields.

// Redact clears every sensitive field in pb.
func Redact(pb proto.Message) {
   // ...
}

This function accepts a proto.Message, an interface type implemented by all generated message types. This type is an alias for one defined in the protoreflect package:

type ProtoMessage interface{
    ProtoReflect() Message
}

To avoid filling up the namespace of generated messages, the interface contains only a single method returning a protoreflect.Message, which provides access to the message contents.

(Why an alias? Because protoreflect.Message has a corresponding method returning the original proto.Message, and we need to avoid an import cycle between the two packages.)

The protoreflect.Message.Range method calls a function for every populated field in a message.

m := pb.ProtoReflect()
m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
    // ...
    return true
})

The range function is called with a protoreflect.FieldDescriptor describing the protocol buffer type of the field, and a protoreflect.Value containing the field value.

The protoreflect.FieldDescriptor.Options method returns the field options as a google.protobuf.FieldOptions message.

opts := fd.Options().(*descriptorpb.FieldOptions)

(Why the type assertion? Since the generated descriptorpb package depends on protoreflect, the protoreflect package can't return the concrete options type without causing an import cycle.)

We can then check the options to see the value of our extension boolean:

if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
    return true // don't redact non-sensitive fields
}

Note that we are looking at the field descriptor here, not the field value. The information we're interested in lies in the protocol buffer type system, not the Go one.

This is also an example of an area where we have simplified the proto package API. The original proto.GetExtension returned both a value and an error. The new proto.GetExtension returns just a value, returning the default value for the field if it is not present. Extension decoding errors are reported at Unmarshal time.

Once we have identified a field that needs redaction, clearing it is simple:

m.Clear(fd)

Putting all the above together, our complete redaction function is:

// Redact clears every sensitive field in pb.
func Redact(pb proto.Message) {
    m := pb.ProtoReflect()
    m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
        opts := fd.Options().(*descriptorpb.FieldOptions)
        if proto.GetExtension(opts, policypb.E_NonSensitive).(bool) {
            return true
        }
        m.Clear(fd)
        return true
    })
}

A more complete implementation might recursively descend into message-valued fields. We hope that this simple example gives a taste of protocol buffer reflection and its uses.

Versions

We call the original version of Go protocol buffers APIv1, and the new one APIv2. Because APIv2 is not backwards compatible with APIv1, we need to use different module paths for each.

(These API versions are not the same as the versions of the protocol buffer language: proto1, proto2, and proto3. APIv1 and APIv2 are concrete implementations in Go that both support the proto2 and proto3 language versions.)

The github.com/golang/protobuf module is APIv1.

The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. (We considered google.golang.org/protobuf/v2, to make it clear that this is the second major version of the API, but settled on the shorter path as being the better choice in the long term.)

We know that not all users will move to a new major version of a package at the same rate. Some will switch quickly; others may remain on the old version indefinitely. Even within a single program, some parts may use one API while others use another. It is essential, therefore, that we continue to support programs that use APIv1.

  • github.com/golang/protobuf@v1.3.4 is the most recent pre-APIv2 version of APIv1.

  • github.com/golang/protobuf@v1.4.0 is a version of APIv1 implemented in terms of APIv2. The API is the same, but the underlying implementation is backed by the new one. This version contains functions to convert between the APIv1 and APIv2 proto.Message interfaces to ease the transition between the two.

  • google.golang.org/protobuf@v1.20.0 is APIv2. This module depends upon github.com/golang/protobuf@v1.4.0, so any program which uses APIv2 will automatically pick a version of APIv1 which integrates with it.

(Why start at version v1.20.0? To provide clarity. We do not anticipate APIv1 to ever reach v1.20.0, so the version number alone should be enough to unambiguously differentiate between APIv1 and APIv2.)

We intend to maintain support for APIv1 indefinitely.

This organization ensures that any given program will use only a single protocol buffer implementation, regardless of which API version it uses. It permits programs to adopt the new API gradually, or not at all, while still gaining the advantages of the new implementation. The principle of minimum version selection means that programs may remain on the old implementation until the maintainers choose to update to the new one (either directly, or by updating a dependency).

Additional features of note

The google.golang.org/protobuf/encoding/protojson package converts protocol buffer messages to and from JSON using the canonical JSON mapping, and fixes a number of issues with the old jsonpb package that were difficult to change without causing problems for existing users.

The google.golang.org/protobuf/types/dynamicpb package provides an implementation of proto.Message for messages whose protocol buffer type is derived at runtime.

The google.golang.org/protobuf/testing/protocmp package provides functions to compare protocol buffer messages with the github.com/google/cmp package.

The google.golang.org/protobuf/compiler/protogen package provides support for writing protocol compiler plugins.

Conclusion

The google.golang.org/protobuf module is a major overhaul of Go's support for protocol buffers, providing first-class support for reflection, custom message implementations, and a cleaned up API surface. We intend to maintain the previous API indefinitely as a wrapper of the new one, allowing users to adopt the new API incrementally at their own pace.

Our goal in this update is to improve upon the benefits of the old API while addressing its shortcomings. As we completed each component of the new implementation, we put it into use within Google's codebase. This incremental rollout has given us confidence in both the usability of the new API and the performance and correctness of the new implementation. We believe it is production ready.

We are excited about this release and hope that it will serve the Go ecosystem for the next ten years and beyond!

Go 1.14 is released

Alex Rakoczy
25 February 2020

Today the Go team is very happy to announce the release of Go 1.14. You can get it from the download page.

Some of the highlights include:

For the complete list of changes and more information about the improvements above, see the Go 1.14 release notes.

We want to thank everyone who contributed to this release by writing code, filing bugs, providing feedback, and/or testing the beta and release candidate. Your contributions and diligence helped to ensure that Go 1.14 is as stable as possible. That said, if you notice any problems, please file an issue.

We hope you enjoy the new release!

Next steps for pkg.go.dev

Julie Qiu
31 January 2020

Introduction

In 2019, we launched go.dev, a new hub for Go developers.

As part of the site, we also launched pkg.go.dev, a central source of information about Go packages and modules. Like godoc.org, pkg.go.dev serves Go documentation. However, it also understands modules and has information about past versions of a package!

Throughout this year, we will be adding features to pkg.go.dev to help our users better understand their dependencies and help them make better decisions around what libraries to import.

Redirecting godoc.org requests to pkg.go.dev

To minimize confusion about which site to use, later this year we are planning to redirect traffic from godoc.org to the corresponding page on pkg.go.dev. We need your help to ensure that pkg.go.dev addresses all of our users' needs. We encourage everyone to begin using pkg.go.dev today for all of their needs and provide feedback.

Your feedback will inform our transition plan, with the goal of making pkg.go.dev our primary source of information and documentation for packages and modules. We’re sure there are things that you want to see on pkg.go.dev, and we want to hear from you about what those features are!

You can share your feedback with us on these channels:

As part of this transition, we will also be discussing plans for API access to pkg.go.dev. We will be posting updates on Go issue 33654.

Frequently asked questions

Since our launch in November, we’ve received tons of great feedback about pkg.go.dev from Go users. For the remainder of this post, we thought it would be helpful to answer some frequently asked questions.

My package doesn’t show up on pkg.go.dev! How do I add it?

We monitor the Go Module Index regularly for new packages to add to pkg.go.dev. If you don’t see a package on pkg.go.dev, you can add it by fetching the module version from proxy.golang.org. See go.dev/about for instructions.

My package has license restrictions. What’s wrong with it?

We understand it can be a frustrating experience to not be able to see the package you want in its entirety on pkg.go.dev. We appreciate your patience as we improve our license detection algorithm.

Since our launch in November, we've made the following improvements:

  • Updated our license policy to include the list of licenses that we detect and recognize
  • Worked with the licensecheck team to improve detection for copyright notices
  • Established a manual review process for special cases

As always, our license policy is at pkg.go.dev/license-policy. If you are having issues, feel free to file an issue on the Go issue tracker, or email go-discovery-feedback@google.com so that we can work with you directly!

Will pkg.go.dev be open-sourced so I can run it at work for my private code?

We understand that corporations with private code want to run a documentation server that provides module support. We want to help meet that need, but we feel we don’t yet understand it as well as we need to.

We’ve heard from users that running the godoc.org server is more complex than it should be, because it is designed for serving at public internet scale instead of just within a company. We believe the current pkg.go.dev server would have the same problem.

We think a new server is more likely to be the right answer for use with private code, instead of exposing every company to the complexity of running the internet-scale pkg.go.dev codebase. In addition to serving documentation, a new server could also serve information to goimports and gopls.

If you want to run such a server, please fill out this 3-5 minute survey to help us better understand your needs. This survey will be available until March 1st, 2020.

We’re excited about the future of pkg.go.dev in 2020, and we hope you are too! We look forward to hearing your feedback and working with the Go community on this transition.

See the index for more articles.