26 March 2018
Eight years ago, the Go team introduced
(which led to
and with it the decentralized, URL-like import paths
that Go developers are familiar with today.
After we released
goinstall, one of the first questions people asked
was how to incorporate version information.
We admitted we didn’t know.
For a long time, we believed that the problem of package versioning
would be best solved by an add-on tool,
and we encouraged people to create one.
The Go community created many tools with different approaches.
Each one helped us all better understand the problem,
but by mid-2015 it was clear that there were now too many solutions.
We needed to adopt a single, official tool.
After a community discussion started at GopherCon in July 2015 and continuing into the fall,
we all believed the answer would be to follow the package versioning approach
exemplified by Rust’s Cargo, with tagged semantic versions,
a manifest, a lock file, and a
SAT solver to decide which versions to use.
Sam Boyer led a team to create Dep, which followed this rough plan,
and which we intended to serve as the model for
go command integration.
But as we learned more about the implications of the Cargo/Dep approach,
it became clear to me that Go would benefit from changing
some of the details, especially concerning backwards compatibility.
The Impact of Compatibility
The most important new feature of Go 1 was not a language feature. It was Go 1’s emphasis on backwards compatibility. Until that point we’d issued stable release snapshots approximately monthly, each with significant incompatible changes. We observed significant acceleration in interest and adoption immediately after the release of Go 1. We believe that the promise of compatibility made developers feel much more comfortable relying on Go for production use and is a key reason that Go is popular today. Since 2013 the Go FAQ has encouraged package developers to provide their own users with similar expectations of compatibility. We call this the import compatibility rule: “If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.”
Independently, semantic versioning has become the de facto standard for describing software versions in many language communities, including the Go community. Using semantic versioning, later versions are expected to be backwards-compatible with earlier versions, but only within a single major version: v1.2.3 must be compatible with v1.2.1 and v1.1.5, but v2.3.4 need not be compatible with any of those.
If we adopt semantic versioning for Go packages,
as most Go developers expect,
then the import compatibility rule requires that
different major versions must use different import paths.
This observation led us to semantic import versioning,
in which versions starting at v2.0.0 include the major
version in the import path:
A year ago I strongly believed that whether to include version numbers in import paths was largely a matter of taste, and I was skeptical that having them was particularly elegant. But the decision turns out to be a matter not of taste but of logic: import compatibility and semantic versioning together require semantic import versioning. When I realized this, the logical necessity surprised me.
I was also surprised to realize that there is a second, independent logical route to semantic import versioning: gradual code repair or partial code upgrades. In a large program, it’s unrealistic to expect all packages in the program to update from v1 to v2 of a particular dependency at the same time. Instead, it must be possible for some of the program to keep using v1 while other parts have upgraded to v2. But then the program’s build, and the program’s final binary, must include both v1 and v2 of the dependency. Giving them the same import path would lead to confusion, violating what we might call the import uniqueness rule: different packages must have different import paths. The only way to have partial code upgrades, import uniqueness, and semantic versioning is to adopt semantic import versioning as well.
It is of course possible to build systems that use semantic versioning without semantic import versioning, but only by giving up either partial code upgrades or import uniqueness. Cargo allows partial code upgrades by giving up import uniqueness: a given import path can have different meanings in different parts of a large build. Dep ensures import uniqueness by giving up partial code upgrades: all packages involved in a large build must find a single agreed-upon version of a given dependency, raising the possibility that large programs will be unbuildable. Cargo is right to insist on partial code upgrades, which are critical to large-scale software development. Dep is equally right to insist on import uniqueness. Complex uses of Go’s current vendoring support can violate import uniqueness. When they have, the resulting problems have been quite challenging for both developers and tools to understand. Deciding between partial code upgrades and import uniqueness requires predicting which will hurt more to give up. Semantic import versioning lets us avoid the choice and keep both instead.
I was also surprised to discover how much import compatibility simplifies version selection, which is the problem of deciding which package versions to use for a given build. The constraints of Cargo and Dep make version selection equivalent to solving Boolean satisfiability, meaning it can be very expensive to determine whether a valid version configuration even exists. And then there may be many valid configurations, with no clear criteria for choosing the “best” one. Relying on import compatibility can instead let Go use a trivial, linear-time algorithm to find the single best configuration, which always exists. This algorithm, which I call minimal version selection, in turn eliminates the need for separate lock and manifest files. It replaces them with a single, short configuration file, edited directly by both developers and tools, that still supports reproducible builds.
Our experience with Dep demonstrates the impact of compatibility. Following the lead of Cargo and earlier systems, we designed Dep to give up import compatibility as part of adopting semantic versioning. I don’t believe we decided this deliberately; we just followed those other systems. The first-hand experience of using Dep helped us better understand exactly how much complexity is created by permitting incompatible import paths. Reviving the import compatibility rule by introducing semantic import versioning eliminates that complexity, leading to a much simpler system.
Progress, a Prototype, and a Proposal
Dep was released in January 2017. Its basic model—code tagged with semantic versions, along with a configuration file that specified dependency requirements—was a clear step forward from most of the Go vendoring tools, and converging on Dep itself was also a clear step forward. I wholeheartedly encouraged its adoption, especially to help developers get used to thinking about Go package versions, both for their own code and their dependencies. While Dep was clearly moving us in the right direction, I had lingering concerns about the complexity devil in the details. I was particularly concerned about Dep lacking support for gradual code upgrades in large programs. Over the course of 2017, I talked to many people, including Sam Boyer and the rest of the package management working group, but none of us could see any clear way to reduce the complexity. (I did find many approaches that added to it.) Approaching the end of the year, it still seemed like SAT solvers and unsatisfiable builds might be the best we could do.
In mid-November, trying once again to work through how Dep could support gradual code upgrades, I realized that our old advice about import compatibility implied semantic import versioning. That seemed like a real breakthrough. I wrote a first draft of what became my semantic import versioning blog post, concluding it by suggesting that Dep adopt the convention. I sent the draft to the people I’d been talking to, and it elicited very strong responses: everyone loved it or hated it. I realized that I needed to work out more of the implications of semantic import versioning before circulating the idea further, and I set out to do that.
In mid-December, I discovered that import compatibility and semantic import versioning together allowed cutting version selection down to minimal version selection. I wrote a basic implementation to be sure I understood it, I spent a while learning the theory behind why it was so simple, and I wrote a draft of the post describing it. Even so, I still wasn’t sure the approach would be practical in a real tool like Dep. It was clear that a prototype was needed.
In January, I started work on a simple
go command wrapper
that implemented semantic import versioning
and minimal version selection.
Trivial tests worked well.
Approaching the end of the month,
my simple wrapper could build Dep,
a real program that made use of many versioned packages.
The wrapper still had no command-line interface—the fact that
it was building Dep was hard-coded in a few string constants—but
the approach was clearly viable.
I spent the first three weeks of February turning the
wrapper into a full versioned
writing drafts of a
blog post series introducing
and discussing them with
Sam Boyer, the package management working group,
and the Go team.
And then I spent the last week of February finally
vgo and the ideas behind it with the whole Go community.
In addition to the core ideas of import compatibility,
semantic import versioning, and minimal version selection,
vgo prototype introduces a number of smaller
but significant changes motivated by eight years of
the new concept of a Go module,
which is a collection of packages versioned as a unit;
verifiable and verified builds;
version-awareness throughout the
enabling work outside
and the elimination of (most)
The result of all of this is the official Go proposal,
which I filed last week.
Even though it might look like a complete implementation,
it’s still just a prototype,
one that we will all need to work together to complete.
You can download and try the
vgo prototype from golang.org/x/vgo,
and you can read the
Tour of Versioned Go
to get a sense of what using
vgo is like.
The Path Forward
The proposal I filed last week is exactly that: an initial proposal. I know there are problems with it that the Go team and I can’t see, because Go developers use Go in many clever ways that we don’t know about. The goal of the proposal feedback process is for us all to work together to identify and address the problems in the current proposal, to make sure that the final implementation that ships in a future Go release works well for as many developers as possible. Please point out problems on the proposal discussion issue. I will keep the discussion summary and FAQ updated as feedback arrives.
For this proposal to succeed, the Go ecosystem as a whole—and in particular today’s major Go projects—will need to adopt the import compatibility rule and semantic import versioning. To make sure that can happen smoothly, we will also be conducting user feedback sessions by video conference with projects that have questions about how to incorporate the new versioning proposal into their code bases or have feedback about their experiences. If you are interested in participating in such a session, please email Steve Francia at firstname.lastname@example.org.
We’re looking forward to (finally!) providing the Go community with a single, official answer
to the question of how to incorporate package versioning into
Thanks to everyone who helped us get this far, and to everyone who will help us going forward.
We hope that, with your help, we can ship something that Go developers will love.
26 February 2018
This post summarizes the result of our 2017 user survey along with commentary and insights. It also draws key comparisons between the results of the 2016 and 2017 survey.
This year we had 6,173 survey respondents, 70% more than the 3,595 we had in the Go 2016 User Survey. In addition, it also had a slightly higher completion rate (84% → 87%) and a higher response rate to most of the questions. We believe that survey length is the main cause of this improvement as the 2017 survey was shortened in response to feedback that the 2016 survey was too long.
We are grateful to everyone who provided their feedback through the survey to help shape the future of Go.
For the first time, more survey respondents say they are paid to write Go than say they write it outside work. This indicates a significant shift in Go's user base and in its acceptance by companies for professional software development.
The areas people who responded to the survey work in is mostly consistent with last year, however, mobile and desktop applications have fallen significantly.
Another important shift: the #1 use of Go is now writing API/RPC services (65%, up 5% over 2016), taking over the top spot from writing CLI tools in Go (63%). Both take full advantage of Go's distinguishing features and are key elements of modern cloud computing. As more companies adopt Go, we expect these two uses of Go to continue to thrive.
Most of the metrics reaffirm things we have learned in prior years. Go programmers still overwhelmingly prefer Go. As more time passes Go users are deepening their experience in Go. While Go has increased its lead among Go developers, the order of language rankings remains quite consistent with last year.
In nearly every question around the usage and perception of Go, Go has demonstrated improvement over our prior survey. Users are happier using Go, and a greater percentage prefer using Go for their next project.
When asked about the biggest challenges to their own personal use of Go, users clearly conveyed that lack of dependency management and lack of generics were their two biggest issues, consistent with 2016. In 2017 we laid a foundation to be able to address these issues. We improved our proposal and development process with the addition of Experience Reports which is enabling the project to gather and obtain feedback critical to making these significant changes. We also made sigificant changes under the hood in how Go obtains, and builds packages. This is foundational work essential to addressing our dependency management needs.
These two issues will continue to be a major focus of the project through 2018.
In this section we asked two new questions. Both center around what developers are doing with Go in a more granular way than we've previously asked. We hope this data will provide insights for the Go project and ecosystem.
Since last year there has been an increase of the percentage of people who identified "Go lacks critical features" as the reason they don't use Go more and a decreased percentage who identified "Go not being an appropriate fit". Other than these changes, the list remains consistent with last year.
Reading the data: This question asked how strongly the respondent agreed or disagreed with the statement. The responses for each statement are displayed as sections of a single bar, from “strongly disagree” in deep red on the left end to “strongly agree” in deep blue on the right end. The bars use the same scale as the rest of the graphs, so they can (and do, especially later in the survey) vary in overall length due to lack of responses.
The ratio after the text compares the number of respondents who agreed (including “somewhat agree” and “strongly agree”) to those who disagreed (including “somewhat disagree” and “strongly disagree”). For example, the ratio of respondents agreeing that they would recommend Go to respondents disagreeing was 19 to 1. The second ratio (within the brackets) is simply a weighted ratio with each somewhat = 1, agree/disagree = 2, and strongly = 4.
Reading the data: This question asked for write-in responses. The bars above show the fraction of surveys mentioning common words or phrases. Only words or phrases that appeared in 20 or more surveys are listed, and meaningless common words or phrases like “the” or “to be” are omitted. The displayed results do overlap: for example, the 402 responses that mentioned “management” do include the 266 listed separately that mentioned “dependency management” and the 79 listed separately that mentioned “package management.” However, nearly or completely redundant shorter entries are omitted: there are not twenty or more surveys that listed “dependency” without mentioning “dependency management,” so there is no separate entry for “dependency.”
Development and deployment
We asked programmers which operating systems they develop Go on; the ratios of their responses remain consistent with last year. 64% of respondents say they use Linux, 49% use MacOS, and 18% use Windows, with multiple choices allowed.
Continuing its explosive growth, VSCode is now the most popular editor among Gophers. IntelliJ/GoLand also saw significant increase in usage. These largely came at the expense of Atom and Submlime Text which saw relative usage drops. This question had a 6% higher response rate from last year.
Survey respondents demonstrated significantly higher satisfaction with Go support in their editors over 2016 with the ratio of satisfied to dissatisfied doubling (9:1 → 18:1). Thank you to everyone who worked on Go editor support for all your hard work.
Go deployment is roughly evenly split between privately managed servers and hosted cloud servers. For Go applications, Google Cloud services saw significant increase over 2016. For Non-Go applications, AWS Lambda saw the largest increase in use.
We asked how strongly people agreed or disagreed with various statements about Go. All questions are repeated from last year with the addition of one new question which we introduced to add further clarifaction around how users are able to both find and use Go libraries.
All responses either indicated a small improvement or are comparable to 2016.
As in 2016, the most commonly requested missing library for Go is one for writing GUIs though the demand is not as pronounced as last year. No other missing library registered a significant number of responses.
The primary sources for finding answers to Go questions are the Go web site, Stack Overflow, and reading source code directly. Stack Overflow showed a small increase from usage over last year.
The primary sources for Go news are still the Go blog, Reddit’s /r/golang and Twitter; like last year, there may be some bias here since these are also how the survey was announced.
The Go Project
59% of respondents expressed interest in contributing in some way to the Go community and projects, up from 55% last year. Respondents also indicated that they felt much more welcome to contribute than in 2016. Unfortunately, respondents indicated only a very tiny improvement in understanding how to contribute. We will be actively working with the community and its leaders to make this a more accessible process.
Respondents showed an increase in agreement that they are confident in the leadership of the Go project (9:1 → 11:1). They also showed a small increase in agreement that the project leadership understands their needs (2.6:1 → 2.8:1) and in agreement that they feel comfortable approaching project leadership with questions and feedback (2.2:1 → 2.4:1). While improvements were made, this continues to be an area of focus for the project and its leadership going forward. We will continue to work to improve our understanding of user needs and approachability.
We tried some new ways to engage with users in 2017 and while progress was made, we are still working on making these solutions scalable for our growing community.
At the end of the survey, we asked some demographic questions.
The country distribution of responses is largely similar to last year with minor fluctuations. Like last year, the distribution of countries is similar to the visits to golang.org, though some Asian countries remain under-represented in the survey.
Perhaps the most significant improvement over 2016 came from the question which asked to what degree do respondents agreed with the statement, "I feel welcome in the Go community". Last year the agreement to disagreement ratio was 15:1. In 2017 this ratio nearly doubled to 25:1.
An important part of a community is making everyone feel welcome, especially people from under-represented demographics. We asked an optional question about identification across a few underrepresented groups. We had a 4% increase in response rate over last year. The percentage of each underrepresented group increased over 2016, some quite significantly.
Like last year, we took the results of the statement “I feel welcome in the Go community” and broke them down by responses to the various underrepresented categories. Like the whole, most of the respondents who identified as underrepresented also felt significantly more welcome in the Go community than in 2016. Respondents who identified as a woman showed the most significant improvement with an increase of over 400% in the ratio of agree:disagree to this statement (3:1 → 13:1). People who identified as ethnically or racially underrepresented had an increase of over 250% (7:1 → 18:1). Like last year, those who identified as not underrepresented still had a much higher percentage of agreement to this statement than those identifying from underrepresented groups.
We are encouraged by this progress and hope that the momentum continues.
The final question on the survey was just for fun: what’s your favorite Go
keyword? Perhaps unsurprisingly, the most popular response was
go, followed by
select, unchanged from last year.
Finally, on behalf of the entire Go project, we are grateful for everyone who has contributed to our project, whether by being a part of our great community, by taking this survey or by taking an interest in Go.
16 February 2018
Happy Friday, happy weekend! Today the Go team is happy to announce the release of Go 1.10. You can get it from the download page.
See the Go 1.10 release notes for all the details.
The most exciting part of this release for many people will probably
be that the
go tool now does
automatic caching of build & test results.
Of course, one of the hundreds of smaller changes may be your favorite.
To celebrate the release, Go User Groups around the world are holding release parties. See if there's one in your area, or feel free to organize one!
Thanks to everyone who contributed to this release and everyone who helped test the Go 1.10 betas and release candidates to ensure a perfect, bug-free final release. However, if you do notice any bugs or unexpected changes not noted in the release notes, be sure to file a bug.
Enjoy the weekend, and enjoy the new release!
P.S. Many of this year's Go conferences are accepting talk proposals this month. We always love to see new speakers and encourage you to think about proposing a talk. For more information, see golang.org/wiki/NewSpeakers.
22 January 2018
We are thrilled to announce that the content on golang.org is now available in mainland China through the name https://golang.google.cn. The growing Go developer community in China can now directly access official documentation, technical articles, and binaries.
The Go community in China is bigger than ever. In 2015, Robert Griesemer visited Shanghai to attend GopherChina, the first Go conference in the country. In the years since, it has become one of the largest Go conferences in the world with over 1200 attendees at their 2017 event. Over the same period, one of the most popular community-built Go forums saw their traffic increase threefold and the number of participants in Go-specific groups on social platforms like WeChat and QQ has grown to over 11,000 people.
We’re excited to provide even more resources for Go developers in China to supplement the excellent material already available to them, but this is just the beginning. We’ll be focusing on making Go more accessible to non-English speakers in 2018, so keep watching this space.
16 November 2017
The Go project wants to hear from you (again)!
Last year we conducted the first ever Go user survey. Thanks to all of you, it was an overwhelming success with over 3500 responses. The survey provided key insights and helped us better plan and prioritize.
We invite you to participate in the 2017 Go User Survey.
The Go User Survey
Who: If you currently use Go, have used Go in the past, or have any interest in the language, please help by sharing your feedback to improve Go for you and your fellow Gophers.
Where: Please take this 15-minute survey by Friday December 8th: 2017 Go User Survey
The survey is anonymous and confidential.
Why: The Go project leadership depends on your feedback to guide the future of the Go project. Your responses will help to understand what's going well and what's not, as well as help us prioritize improvements for the Go language, libraries and tools.
A few weeks after the survey closes, we will publish the anonymous aggregate results to the Go blog. See the 2016 Go user survey results to learn what insights were gained from last year's survey.
Spread the word!
Please help us spread the word by sharing this post on your social network feeds, at meetups, around your office and in other communities.
See the index for more articles.