The Go Blog

Contribution Workshop

9 August 2017

Event Overview

by Steve

During the community day at GopherCon, the Go team held two workshops where we worked with people to help them make their first contribution to the Go project. This was the first time the Go project has ever attempted anything like this. We had about 140 participants and about 35 people who volunteered as mentors. Mentors not only received warm fuzzy feelings for helping others, but also a very stylish Go Mentor trucker hat. We had contributors of all ages and experience levels coming from North and South America, Africa, Europe, Asia, and Australia. It was truly a worldwide effort of Gophers coming together at GopherCon.

One of our reasons for running the workshop was for it to act as a forcing function to make us improve our contributor experience. In preparation for the workshop, we rewrote our contributor guide, including adding a "troubleshooting" section and built a tool go-contrib-init, which automated the process of setting up a development environment to be able to contribute to Go.

For the workshop itself, we developed a presentation "Contributing to Go," and a dashboard / scoreboard that was presented during the event. The scoreboard was designed to encourage us all to work together towards a common goal of seeing our collective score increase. Participants added 1, 2 or 3 points to the total score when they performed actions like registering an account, making a change list (also known as a CL, similar to a pull request), amending a CL, or submitting a CL.

Brad Fitzpatrick, who stayed home from GopherCon this year, was ready and waiting to review all CLs submitted. He was so quick to review that many people thought he was an automated bot. Internally our team is now calling him "BradBot" mostly because we are in awe and a bit jealous.

Impact

We had a total of 65 CLs submitted from the people who participated in the workshop (within a week of the workshop). Of these, 44 were from contributors who had never previously contributed to any of the repos in the Go project. Half (22) of these contributions were already merged. Many of the others are waiting on the codebase to thaw as we are in the middle of a freeze for the upcoming 1.9 release. In addition to CLs, many contributed to the project in the form of bug reports, gardening tasks, and other types of contributions.

The most common type of contribution was an example function to be used in the documentation. The Go User survey identified that our documentation was significantly lacking examples. In the presentation, we asked users to find a package they loved and to add an example. In the Go project, examples are written as code in Go files (with specific naming) and the `go doc` tool displays them along side the documentation. This is a perfect first contribution as it's something that can be merged during a freeze, it's of critical importance to our users, and it's an addition that has a relatively narrow scope.

One of the examples added is that of creating a Stringer, one of the more widely used interfaces in Go. CL 49270

In addition to examples, many people contributed critical bug fixes including:

Some people even surprised us by arriving with a bug in mind that they wanted to fix. Nikhita arrived ready to tackle issue #20786 and she did submitting CL 48871, after which she tweeted:

Not only were some great improvements made, but most importantly, we narrowed the gap between the core Go team and the broader community members. Many people on the Go team remarked that the community members were teaching them things about the Go project. People in the community (in person, and on Twitter) remarked that felt welcome to participate in the project.

Future

The event was successful well beyond our expectations. Sameer Ajmani, Go team manager said, "The contributor workshop was incredibly fun and educational–for the Go team. We cringed as users hit the rough edges in our process, and celebrated when they got up on the dashboard. The cheer when the group score hit 1000 was awesome."

We are looking into ways to make this workshop easier to run for future events (like meetups and conferences). Our biggest challenge is providing enough mentorship so that users feel supported. If you have any ideas or would like to help with this process please let me know.

I've asked a few participants of the event to share their experiences below:

My Contribution Experience

by Cassandra

When I heard about the go-contrib workshop I was very excited and then I was extremely intimidated. I was encouraged by a member of the Go team to participate, so I thought what the heck.

As I walked into the room (let's be real, I ran into the room because I was running late) I was pleased to see the room was jam-packed. I looked around for people in Gopher caps, which was the main indicator they were teachers. I sat down at one of the 16 round tables that had two hats and three non-hats. Brought up my screen and was ready to roll…

Jess Frazelle stood up and started the presentation and provided the group with a link to make it easy to follow.

The murmurs grew from a deep undercurrent to a resounding melody of voices, people were getting their computers set up with Go, they were skipping ahead to make sure their GOPATH was set, and were… wait what's Gerrit?

Most of us had to get a little intro to Gerrit. I had no clue what it was, but luckily there was a handy slide. Jess explained that it was an alternative to GitHub with slightly more advanced code review tools. We then went through GitHub vs Geritt terminology, so we had better understanding of the process.

Ok, now it was time to become a freaking* Go contributor*.

To make this more exciting than it already is, the Go team set up a game where we could track as a group how many points we could rack up based on the Gerrit score system.

Seeing your name pop up on the board and listening to everyone's excited was intoxicating. It also invoked a sense of teamwork that lead to a feeling of inclusion and feeling like you were truly a part of the Go community.

In 6 steps a room of around 80 people were able to learn how to contribute to go within an hour. That's a feat!

It wasn't nearly as difficult as I anticipated and it wasn't out of scope for a total newbie. It fostered a sense of community in an active and tangible way as well as a sense of inclusion in the illustrious process of Go contributions.

I'd personally like to thank the Go Team, the Gopher mentors in hats, and my fellow participants for making it one of my most memorable moments at GopherCon.

My Contribution Experience

by Matt

I've always found programming languages to be intimidating. It's the code that enables me to write code. Surely smarter people than me should be working on it. It feels too distant to touch and too significant for me to edit. So when the opportunity to join a workshop to contribute to my new favorite programming language came up, I was as excited to see how I could help. A month later, I'm now certain that anyone and everyone can (and should) contribute back to Go.

Here are my very verbose steps to go from 0 to 2 contributions to Go:

The Setup

Given Go's use of Gerrit, I started by setting up my environment for it. Jess Frazzelle's guide is a great place to start to not miss a step.

The real fun starts when you clone the Go repo. Ironically, you don't hack on Go under $GOPATH, so I put it in my other workspace (which is ~/Develop).

cd $DEV # That's my source code folder outside of $GOPATH
git clone --depth 1 https://go.googlesource.com/go

Then install the helper tool, go-contrib-init:

go get -u golang.org/x/tools/cmd/go-contrib-init

Now you can run go-contrib-init from the folder we cloned above and see whether or not we're ready to contribute. If you're following along you're definitely not ready yet.

Next, install codereview so you can participate in a Gerrit code review:

go get -u golang.org/x/review/git-codereview

This package includes `git change` and `git mail` which will replace your normal workflow for `git commit` and `git push` respectively.

Okay, installations are out of the way. Now setup your Gerrit account here, then sign the CLA appropriate for you (more on choosing a CLA here. For me it was a personal one for all Google projects. You can see all CLAs you've signed at cla.developers.google.com/clas).

AND BAM. You're good (to go)! But where to contribute?

Contributing

In workshop sent us into the scratch repository, which is a safe place to fool around in order to master the workflow:

cd $(go env GOPATH)/src/golang.org/x
git clone --depth 1 [[https://go.googlesource.com/scratch][go.googlesource.com/scratch]]

First stop is to cd in and run go-contrib-init to make sure you're ready to contribute:

go-contrib-init
All good. Happy hacking!

From there, I made a folder, did a `git add -u` then took `git change` for a spin. It has what the Go team call the "magic hash", which is the one line you shouldn't touch. Other than that, it feels just like `git commit`. Once I got the commit message matching the format of `package: description` (description begins with a lowercase), I used `git mail` to send it over to Gerrit.

Two good to know things: `git change` also works like `git commit --amend`, so if you need to update your patch you can add then change and it will all link to the same patch. Secondly, you can always review your patch from your personal Gerrit dashboard.

After a few back and forths, I officially had a contribute to Go! And if Jaana is right, it might be the first with emojis :+1:.

Contributing, For Real

The scratch repo is fun and all, but there's a ton of ways to get into the depths of Go's packages and give back. It's at this point where I cruised around the many packages available to see what was available and interesting to me. And by "cruised around" I mean attempted to find a list of packages, then went to my source code to see what's around under the go/src/ folder:

I decided to see what I can do in the regexp package, maybe out of love and fear of regex. Here's where I switched to the website's view of the package (it's good to know that each standard package can be found at https://godoc.org/$PACKAGENAME). In there I noticed that QuoteMeta was missing the same level of detailed examples other functions have (and I could use the practice using Gerrit).

I started looking at go/src/regexp to try to find where to add examples and I got lost pretty quickly. Lucky for me, Francesc was around that day. He walked me through how all examples are actually in-line tests in a `example_test.go file. They follow the format of test cases followed by "Output" commented out and then the answers to the tests. For example:

func ExampleRegexp_FindString() {
    re := regexp.MustCompile("fo.?")
    fmt.Printf("%q\n", re.FindString("seafood"))
    fmt.Printf("%q\n", re.FindString("meat"))
    // Output:
    // "foo"
    // ""
}

Kind of cool, right?? I followed Francesc's lead and added a function ExampleQuoteMeta and added a few I thought would be helpful. From there it's a `git change` and `git mail` to Gerrit!

I have to say that Steve Francia challenged me to "find something that isn't an open issue and fix it," so I included some documentation changes for QuoteMeta in my patch. It's going to be open for a bit longer given the additional scope, but I think it's worth it on this one.

I can hear your question already: how did I verify it worked? Well it wasn't easy to be honest. Running `go test example_test.go -run QuoteMeta -v` won't do it since we're working outside of our $GOPATH. I struggled to figure it out until Kale Blakenship wrote this awesome post on testing in Go. Bookmark this one for later.

You can see my completed contribution here. What I also hope you see is how simple it is to get into the flow of contributing. If you're like me, you'll be good finding a small typo or missing example in the docs to start to get used to the `git codereview` workflow. After that, you'll be ready to find an open issue, ideally one tagged for an upcoming release, and give it a go. No matter what you choose to do, definitely go forth and do it. The Go team proved to me just how much they care about helping us all contribute back. I can't wait for my next `git mail`.

My Mentorship Experience

by Dmitri

I was looking forward to participating in the Contribution Workshop event as a mentor. I had high expectations for the event, and thought it was a great idea before it started.

I made my first contribution to Go on May 10th, 2014. I remember it was about four months from the moment I wanted to contribute, until that day, when I actually sent my first CL. It took that long to build up the courage and fully commit to figuring out the process. I was an experienced software engineer at the time. Despite that, the Go contribution process felt alien—being unlike all other processes I was already familiar with—and therefore seemed intimidating. It was well documented though, so I knew it would be just a matter of finding the time, sitting down, and doing it. The "unknown" factor kept me from giving it a shot.

After a few months passed, I thought "enough is enough," and decided to dedicate an entire day of an upcoming weekend to figuring out the process. I set aside all of Saturday for doing one thing: sending my first CL to Go. I opened up the Contribution Guide and started following all the steps, from the very top. Within an hour, I was done. I had send my first CL. I was both in awe and shock. In awe, because I had finally sent a contribution to Go, and it was accepted! In shock, because, why did I wait so long to finally do this? Following the steps in the Contribution Guide was very easy, and the entire process went completely smoothly. If only someone had told me that I'd be done within an hour and nothing would go wrong, I would've done it much sooner!

Which brings me to this event and why I thought it was such a good idea. For anyone who ever wanted to contribute to Go, but felt daunted by the unfamiliar and seemingly lengthy process (like I was during those four months), this was their chance! Not only is it easy to commit to figuring it out by attending the event, but also the Go team and helpful volunteer mentors would be there to help you along the way.

Despite the already high expectations I had for the event, my expectations were exceeded. For one, the Go team had prepared really well and invested a lot in making the event that much more enjoyable for everyone. There was a very fun presentation that went over all the contributing steps quickly. There was a dashboard made for the event, where everyone's successfully completed steps were rewarded with points towards a global score. That made it into a very collaborative and social event! Finally, and most importantly, they were Go team members like Brad Fitzpatrick behind the scenes, helping review CLs promptly! That meant the CLs that were submitted received reviews quickly, with actionable next steps, so everyone could move forward and learn more.

I originally anticipated the event to be somewhat dull, in that the contribution steps are extremely simple to follow. However, I found that wasn't always the case, and I was able to use my expertise in Go to help out people who got stuck in various unexpected places. It turns out, the real world is filled with edge cases. For instance, someone had two git emails, one personal and another for work. There was a delay with signing the CLA for the work email, so they tried to use their personal email instead. That meant each commit had to be amended to use the right email, something the tools didn't take into account. (Luckily, there is a troubleshooting section in the contribution guide covering this exact issue!) There were other subtle mistakes or environment misconfiguration that some people ran into, because having more than one Go installation was a bit unusual. Sometimes, the GOROOT environment variable had to be explicitly set, temporarily, to get godoc to show changes in the right standard library (I was tongue-in-cheek looking over my shoulder to check for Dave Cheney as I uttered those words).

Overall, I oversaw a few new gophers make their first Go contributions. They sent the CLs, responded to review feedback, made edits, iterated until everyone was happy, and eventually saw their first Go contributions get merged to master! It was very rewarding to see the happiness on their faces, because the joy of making one's first contribution is something I can relate to myself. It was also great to be able to help them out, and explain tricky situations that they sometimes found themselves. From what I can tell, many happy gophers walked away from the event, myself included!

Photos from the event

Photos by Sameer Ajmani & Steve Francia

By Steve Francia, Cassandra Salisbury, Matt Broberg, Dmitri Shuralyov

Contributors Summit

3 August 2017

Introduction

The day before GopherCon, a group of Go team members and contributors gathered in Denver to discuss and plan for the future of the Go project. This was the first ever event of its kind, a major milestone for the Go project. The event comprised a morning session revolving around focused discussions on a theme, and an afternoon session made up of round table discussions in small break-out groups.

Compiler and runtime

The compiler and runtime session started out with a discussion about refactoring gc and related tools into importable packages. This would reduce overhead in the core tools and in IDEs which could embed the compiler themselves to do quick syntax checking. Code could also be compiled entirely in memory, which is useful in environments that don't provide a filesystem, or to run tests continually while you develop to get a live report of breakages. More discussion about whether or not to pursue this line of work will most likely be brought up on the mailing lists in the future.

There was also a great deal of discussion around bridging the gap between optimized assembly code and Go. Most crypto code in Go is written in assembly for performance reasons; this makes it hard to debug, maintain, and read. Furthermore, once you've ventured into writing assembly, you often can't call back into Go, limiting code reuse. A rewrite in Go would make maintenance easier. Adding processor intrinsics and better support for 128-bit math would improve Go's crypto performance. It was proposed that the new math/bits package coming in 1.9 could be expanded for this purpose.

Not being all that familiar with the development of the compiler and runtime, this for me was one of the more interesting sessions of the day. I learned a lot about the current state of the world, the problems, and where people want to go from here.

Dependency management

After a quick update from the dep team on the status of the project, the dependency management session gravitated towards how the Go world will work once dep (or something dep-like) becomes the primary means of package management. Work to make Go easier to get started with and make dep easier to use has already started. In Go 1.8, a default value for GOPATH was introduced, meaning users will only have to add Go's bin directory to their $PATH before they can get started with dep.

Another future usability improvement that dep might enable, is allowing Go to work from any directory (not just a workspace in the GOPATH), so that people can use the directory structures and workflows they're used to using with other languages. It may also be possible to make `go install` easier in the future by guiding users through the process of adding the bin directory to their path, or even automating the process. There are many good options for making the Go tooling easier to use, and discussion will likely continue on the mailing lists.

The standard library

The discussions we had around the future of the Go language are mostly covered in Russ Cox's blog post: Toward Go 2, so let's move on to the standard library session.

As a contributor to the standard library and subrepos, this session was particularly interesting to me. What goes in the standard library and subrepos, and how much it can change, is a topic that isn't well defined. It can be hard on the Go team to maintain a huge number of packages when they may or may not have anyone with specific expertise in the subject matter. To make critical fixes to packages in the standard library, one must wait 6 months for a new version of Go to ship (or a point release has to be shipped in the case of security issues, which drains team resources). Better dependency management may facilitate the migration of some packages out of the standard library and into their own projects with their own release schedules.

There was also some discussion about things that are difficult to achieve with the interfaces in the standard library. For instance, it would be nice if io.Reader accepted a context so that blocking read operations could be canceled.

More experience reports are necessary before we can determine what will change in the standard library.

Tooling and editors

A language server for editors to use was a hot topic in the tooling session, with a number of people advocating for IDE and tool developers to adopt a common "Go Language Server" to index and display information about code and packages. Microsoft's Language Server Protocol was suggested as a good starting point because of its wide support in editors and IDEs.

Jaana Burcu Dogan also discussed her work on distributed tracing and how information about runtime events could be made easier to acquire and attached to traces. Having a standard "counter" API to report statistics was proposed, but specific experience reports from the community will be required before such an API can be designed.

The contributor experience

The final session of the day was on the contributor experience. The first discussion was all about how the current Gerrit workflow could be made easier for new contributors which has already resulted in improvements to the documentation for several repos, and influenced the new contributors workshop a few days later!

Making it easier to find tasks to work on, empowering users to perform gardening tasks on the issue tracker, and making it easier to find reviewers were also considered. Hopefully we'll see improvements to these and many more areas of the contribution process in the coming weeks and months!

Breakout sessions

In the afternoon, participants broke out into smaller groups to have more in-depth discussions about some of the topics from the morning session. These discussions had more specific goals. For example, one group worked on identifying the useful parts of an experience report and a list of existing literature documenting Go user experiences, resulting in the experience report wiki page.

Another group considered the future of errors in Go. Many Go users are initially confused by, or don't understand the fact that error is an interface, and it can be difficult to attach more information to errors without masking sentinel errors such as io.EOF. The breakout session discussed specific ways it might be possible to fix some of these issues in upcoming Go releases, but also ways error handling could be improved in Go 2.

Community

Outside of the technical discussions, the summit also provided an opportunity for a group of people from all over the world who often talk and work together to meet in person, in many cases for the first time. There is no substitute for a little face-to-face time to build a sense of mutual respect and comradeship, which is critical when a diverse group with different backgrounds and ideas needs to come together to work in a single community. During the breaks, Go team members dispersed themselves among the contributors for discussions both about Go and a little general socialization, which really helped to put faces to the names that review our code every day.

As Russ discussed in Toward Go 2, communicating effectively requires knowing your audience. Having a broad sample of Go contributors in a room together helped us all to understand the Go audience better and start many productive discussions about the future of Go. Going forward, we hope to do more frequent events like this to facilitate discourse and a sense of community.

Photos by Steve Francia

By Sam Whited

Toward Go 2

13 July 2017

Introduction

[This is the text of my talk today at Gophercon 2017, asking for the entire Go community's help as we discuss and plan Go 2. We will add a link to the video when it becomes available.]

On September 25, 2007, after Rob Pike, Robert Griesemer, and Ken Thompson had been discussing a new programming language for a few days, Rob suggested the name “Go.”

The next year, Ian Lance Taylor and I joined the team, and together the five of us built two compilers and a standard library, leading up to the open-source release on November 10, 2009.

For the next two years, with the help of the new Go open source community, we experimented with changes large and small, refining Go and leading to the plan for Go 1, proposed on October 5, 2011.

With more help from the Go community, we revised and implemented that plan, eventually releasing Go 1 on March 28, 2012.

The release of Go 1 marked the culmination of nearly five years of creative, frenetic effort that took us from a name and a list of ideas to a stable, production language. It also marked an explicit shift from change and churn to stability.

In the years leading to Go 1, we changed Go and broke everyone's Go programs nearly every week. We understood that this was keeping Go from use in production settings, where programs could not be rewritten weekly to keep up with language changes. As the blog post announcing Go 1 says, the driving motivation was to provide a stable foundation for creating reliable products, projects, and publications (blogs, tutorials, conference talks, and books), to make users confident that their programs would continue to compile and run without change for years to come.

After Go 1 was released, we knew that we needed to spend time using Go in the production environments it was designed for. We shifted explicitly away from making language changes toward using Go in our own projects and improving the implementation: we ported Go to many new systems, we rewrote nearly every performance-critical piece to make Go run more efficiently, and we added key tools like the race detector.

Now we have five years of experience using Go to build large, production-quality systems. We have developed a sense of what works and what does not. Now it is time to begin the next step in Go's evolution and growth, to plan the future of Go. I'm here today to ask all of you in the Go community, whether you're in the audience at GopherCon or watching on video or reading the Go blog later today, to work with us as we plan and implement Go 2.

In the rest of this talk, I'm going to explain our goals for Go 2; our constraints and limitations; the overall process; the importance of writing about our experiences using Go, especially as they relate to problems we might try to solve; the possible kinds of solutions; how we will deliver Go 2; and how all of you can help.

Goals

The goals we have for Go today are the same as in 2007. We want to make programmers more effective at managing two kinds of scale: production scale, especially concurrent systems interacting with many other servers, exemplified today by cloud software; and development scale, especially large codebases worked on by many engineers coordinating only loosely, exemplified today by modern open-source development.

These kinds of scale show up at companies of all sizes. Even a five-person startup may use large cloud-based API services provided by other companies and use more open-source software than software they write themselves. Production scale and development scale are just as relevant at that startup as they are at Google.

Our goal for Go 2 is to fix the most significant ways Go fails to scale.

(For more about these goals, see Rob Pike's 2012 article “Go at Google: Language Design in the Service of Software Engineering” and my GopherCon 2015 talk “Go, Open Source, Community.”)

Constraints

The goals for Go have not changed since the beginning, but the constraints on Go certainly have. The most important constraint is existing Go usage. We estimate that there are at least half a million Go developers worldwide, which means there are millions of Go source files and at least a billion of lines of Go code. Those programmers and that source code represent Go's success, but they are also the main constraint on Go 2.

Go 2 must bring along all those developers. We must ask them to unlearn old habits and learn new ones only when the reward is great. For example, before Go 1, the method implemented by error types was named String. In Go 1, we renamed it Error, to distinguish error types from other types that can format themselves. The other day I was implementing an error type, and without thinking I named its method String instead of Error, which of course did not compile. After five years I still have not completely unlearned the old way. That kind of clarifying renaming was an important change to make in Go 1 but would be too disruptive for Go 2 without a very good reason.

Go 2 must also bring along all the existing Go 1 source code. We must not split the Go ecosystem. Mixed programs, in which packages written in Go 2 import packages written in Go 1 and vice versa, must work effortlessly during a transition period of multiple years. We'll have to figure out exactly how to do that; automated tooling like go fix will certainly play a part.

To minimize disruption, each change will require careful thought, planning, and tooling, which in turn limits the number of changes we can make. Maybe we can do two or three, certainly not more than five.

I'm not counting minor housekeeping changes like maybe allowing identifiers in more spoken languages or adding binary integer literals. Minor changes like these are also important, but they are easier to get right. I'm focusing today on possible major changes, such as additional support for error handling, or introducing immutable or read-only values, or adding some form of generics, or other important topics not yet suggested. We can do only a few of those major changes. We will have to choose carefully.

Process

That raises an important question. What is the process for developing Go?

In the early days of Go, when there were just five of us, we worked in a pair of adjacent shared offices separated by a glass wall. It was easy to pull everyone into one office to discuss some problem and then go back to our desks to implement a solution. When some wrinkle arose during the implementation, it was easy to gather everyone again. Rob and Robert's office had a small couch and a whiteboard, so typically one of us went in and started writing an example on the board. Usually by the time the example was up, everyone else had reached a good stopping point in their own work and was ready to sit down and discuss it. That informality obviously doesn't scale to the global Go community of today.

Part of the work since the open-source release of Go has been porting our informal process into the more formal world of mailing lists and issue trackers and half a million users, but I don't think we've ever explicitly described our overall process. It's possible we never consciously thought about it. Looking back, though, I think this is the basic outline of our work on Go, the process we've been following since the first prototype was running.

Step 1 is to use Go, to accumulate experience with it.

Step 2 is to identify a problem with Go that might need solving and to articulate it, to explain it to others, to write it down.

Step 3 is to propose a solution to that problem, discuss it with others, and revise the solution based on that discussion.

Step 4 is to implement the solution, evaluate it, and refine it based on that evaluation.

Finally, step 5 is to ship the solution, adding it to the language, or the library, or the set of tools that people use from day to day.

The same person does not have to do all these steps for a particular change. In fact, usually many people collaborate on any given step, and many solutions may be proposed for a single problem. Also, at any point we may realize we don’t want to go further with a particular idea and circle back to an earlier step.

Although I don't believe we've ever talked about this process as a whole, we have explained parts of it. In 2012, when we released Go 1 and said that it was time now to use Go and stop changing it, we were explaining step 1. In 2015, when we introduced the Go change proposal process, we were explaining steps 3, 4, and 5. But we've never explained step 2 in detail, so I'd like to do that now.

(For more about the development of Go 1 and the shift away from language changes, see Rob Pike and Andrew Gerrand's OSCON 2012 talk “The Path to Go 1.” For more about the proposal process, see Andrew Gerrand's GopherCon 2015 talk “How Go was Made” and the proposal process documentation.)

Explaining Problems

There are two parts to explaining a problem. The first part—the easier part—is stating exactly what the problem is. We developers are decently good at this. After all, every test we write is a statement of a problem to be solved, in language so precise that even a computer can understand it. The second part—the harder part—is describing the significance of the problem well enough that everyone can understand why we should spend time solving it and maintaining a solution. In contrast to stating a problem precisely, we don't need to describe a problem's significance very often, and we're not nearly as good at it. Computers never ask us “why is this test case important? Are you sure this is the problem you need to solve? Is solving this problem the most important thing you can be doing?” Maybe they will someday, but not today.

Let's look at an old example from 2011. Here is what I wrote about renaming os.Error to error.Value while we were planning Go 1.

It begins with a precise, one-line statement of the problem: in very low-level libraries everything imports "os" for os.Error. Then there are five lines, which I've underlined here, devoted to describing the significance of the problem: the packages that "os" uses cannot themselves present errors in their APIs, and other packages depend on "os" for reasons having nothing to do with operating system services.

Do these five lines convince you that this problem is significant? It depends on how well you can fill in the context I've left out: being understood requires anticipating what others need to know. For my audience at the time—the ten other people on the Go team at Google who were reading that document—those fifty words were enough. To present the same problem to the audience at GothamGo last fall—an audience with much more varied backgrounds and areas of expertise—I needed to provide more context, and I used about two hundred words, along with real code examples and a diagram. It is a fact of today's worldwide Go community that describing the significance of any problem requires adding context, especially illustrated by concrete examples, that you would leave out when talking to coworkers.

Convincing others that a problem is significant is an essential step. When a problem appears insignificant, almost every solution will seem too expensive. But for a significant problem, there are usually many solutions of reasonable cost. When we disagree about whether to adopt a particular solution, we're often actually disagreeing about the significance of the problem being solved. This is so important that I want to look at two recent examples that show this clearly, at least in hindsight.

Example: Leap seconds

My first example is about time.

Suppose you want to time how long an event takes. You write down the start time, run the event, write down the end time, and then subtract the start time from the end time. If the event took ten milliseconds, the subtraction gives a result of ten milliseconds, perhaps plus or minus a small measurement error.

start := time.Now()       // 3:04:05.000
event()
end := time.Now()         // 3:04:05.010

elapsed := end.Sub(start) // 10 ms

This obvious procedure can fail during a leap second. When our clocks are not quite in sync with the daily rotation of the Earth, a leap second—officially 11:59pm and 60 seconds—is inserted just before midnight. Unlike leap years, leap seconds follow no predictable pattern, which makes them hard to fit into programs and APIs. Instead of trying to represent the occasional 61-second minute, operating systems typically implement a leap second by turning the clock back one second just before what would have been midnight, so that 11:59pm and 59 seconds happens twice. This clock reset makes time appear to move backward, so that our ten-millisecond event might be timed as taking negative 990 milliseconds.

start := time.Now()       // 11:59:59.995
event()
end := time.Now()         // 11:59:59.005 (really 11:59:60.005)

elapsed := end.Sub(start) // –990 ms

Because the time-of-day clock is inaccurate for timing events across clock resets like this, operating systems now provide a second clock, the monotonic clock, which has no absolute meaning but counts seconds and is never reset.

Except during the odd clock reset, the monotonic clock is no better than the time-of-day clock, and the time-of-day clock has the added benefit of being useful for telling time, so for simplicity Go 1’s time APIs expose only the time-of-day clock.

In October 2015, a bug report noted that Go programs could not time events correctly across clock resets, especially a typical leap second. The suggested fix was also the original issue title: “add a new API to access a monotonic clock source.” I argued that this problem was not significant enough to justify new API. A few months earlier, for the mid-2015 leap second, Akamai, Amazon, and Google had slowed their clocks a tiny amount for the entire day, absorbing the extra second without turning their clocks backward. It seemed like eventual widespread adoption of this “leap smear” approach would eliminate leap-second clock resets as a problem on production systems. In contrast, adding new API to Go would add new problems: we would have to explain the two kinds of clocks, educate users about when to use each, and convert many lines of existing code, all for an issue that rarely occurred and might plausibly go away on its own.

We did what we always do when there's a problem without a clear solution: we waited. Waiting gives us more time to add experience and understanding of the problem and also more time to find a good solution. In this case, waiting added to our understanding of the significance of the problem, in the form of a thankfully minor outage at Cloudflare. Their Go code timed DNS requests during the end-of-2016 leap second as taking around negative 990 milliseconds, which caused simultaneous panics across their servers, breaking 0.2% of DNS queries at peak.

Cloudflare is exactly the kind of cloud system Go was intended for, and they had a production outage based on Go not being able to time events correctly. Then, and this is the key point, Cloudflare reported their experience in a blog post by John Graham-Cumming titled “How and why the leap second affected Cloudflare DNS.” By sharing concrete details of their experience with Go in production, John and Cloudflare helped us understand that the problem of accurate timing across leap second clock resets was too significant to leave unfixed. Two months after that article was published, we had designed and implemented a solution that will ship in Go 1.9 (and in fact we did it with no new API).

Example: Alias declarations

My second example is support for alias declarations in Go.

Over the past few years, Google has established a team focused on large-scale code changes, meaning API migration and bug fixes applied across our codebase of millions of source files and billions of lines of code written in C++, Go, Java, Python, and other languages. One thing I've learned from that team's work is the importance, when changing an API from using one name to another, of being able to update client code in multiple steps, not all at once. To do this, it must be possible to write a declaration forwarding uses of the old name to the new name. C++ has #define, typedef, and using declarations to enable this forwarding, but Go has nothing. Of course, one of Go's goals is to scale well to large codebases, and as the amount of Go code at Google grew, it became clear both that we needed some kind of forwarding mechanism and also that other projects and companies would run into this problem as their Go codebases grew.

In March 2016, I started talking with Robert Griesemer and Rob Pike about how Go might handle gradual codebase updates, and we arrived at alias declarations, which are exactly the needed forwarding mechanism. At this point, I felt very good about the way Go was evolving. We'd talked about aliases since the early days of Go—in fact, the first spec draft has an example using alias declarations—but each time we'd discussed aliases, and later type aliases, we had no clear use case for them, so we left them out. Now we were proposing to add aliases not because they were an elegant concept but because they solved a significant practical problem with Go meeting its goal of scalable software development. I hoped this would serve as a model for future changes to Go.

Later in the spring, Robert and Rob wrote a proposal, and Robert presented it in a Gophercon 2016 lightning talk. The next few months did not go smoothly, and they were definitely not a model for future changes to Go. One of the many lessons we learned was the importance of describing the significance of a problem.

A minute ago, I explained the problem to you, giving some background about how it can arise and why, but with no concrete examples that might help you evaluate whether the problem might affect you at some point. Last summer’s proposal and the lightning talk gave an abstract example, involving packages C, L, L1, and C1 through Cn, but no concrete examples that developers could relate to. As a result, most of the feedback from the community was based on the idea that aliases only solved a problem for Google, not for everyone else.

Just as we at Google did not at first understand the significance of handling leap second time resets correctly, we did not effectively convey to the broader Go community the significance of handling gradual code migration and repair during large-scale changes.

In the fall we started over. I gave a talk and wrote an article presenting the problem using multiple concrete examples drawn from open source codebases, showing how this problem arises everywhere, not just inside Google. Now that more people understood the problem and could see its significance, we had a productive discussion about what kind of solution would be best. The outcome is that type aliases will be included in Go 1.9 and will help Go scale to ever-larger codebases.

Experience reports

The lesson here is that it is difficult but essential to describe the significance of a problem in a way that someone working in a different environment can understand. To discuss major changes to Go as a community, we will need to pay particular attention to describing the significance of any problem we want to solve. The clearest way to do that is by showing how the problem affects real programs and real production systems, like in Cloudflare's blog post and in my refactoring article.

Experience reports like these turn an abstract problem into a concrete one and help us understand its significance. They also serve as test cases: any proposed solution can be evaluated by examining its effect on the actual, real-world problems the reports describe.

For example, I've been examining generics recently, but I don't have in my mind a clear picture of the detailed, concrete problems that Go users need generics to solve. As a result, I can't answer a design question like whether to support generic methods, which is to say methods that are parameterized separately from the receiver. If we had a large set of real-world use cases, we could begin to answer a question like this by examining the significant ones.

As another example, I’ve seen proposals to extend the error interface in various ways, but I haven't seen any experience reports showing how large Go programs attempt to understand and handle errors at all, much less showing how the current error interface hinders those attempts. These reports would help us all better understand the details and significance of the problem, which we must do before solving it.

I could go on. Every major potential change to Go should be motivated by one or more experience reports documenting how people use Go today and why that's not working well enough. For the obvious major changes we might consider for Go, I'm not aware of many such reports, especially not reports illustrated with real-world examples.

These reports are the raw material for the Go 2 proposal process, and we need all of you to write them, to help us understand your experiences with Go. There are half a million of you, working in a broad range of environments, and not that many of us. Write a post on your own blog, or write a Medium post, or write a Github Gist (add a .md file extension for Markdown), or write a Google doc, or use any other publishing mechanism you like. After you've posted, please add the post to our new wiki page, golang.org/wiki/ExperienceReports.

Solutions

Now that we know how we're going to identify and explain problems that need to be solved, I want to note briefly that not all problems are best solved by language changes, and that's fine.

One problem we might want to solve is that computers can often compute additional results during basic arithmetic operations, but Go does not provide direct access to those results. In 2013, Robert proposed that we might extend the idea of two-result (“comma-ok”) expressions to basic arithmetic. For example, if x and y are, say, uint32 values, lo, hi = x * y would return not only the usual low 32 bits but also the high 32 bits of the product. This problem didn't seem particularly significant, so we recorded the potential solution but didn't implement it. We waited.

More recently, we designed for Go 1.9 a math/bits package that contains various bit manipulation functions:

package bits // import "math/bits"

func LeadingZeros32(x uint32) int
func Len32(x uint32) int
func OnesCount32(x uint32) int
func Reverse32(x uint32) uint32
func ReverseBytes32(x uint32) uint32
func RotateLeft32(x uint32, k int) uint32
func TrailingZeros32(x uint32) int
...

The package has good Go implementations of each function, but the compilers also substitute special hardware instructions when available. Based on this experience with math/bits, both Robert and I now believe that making the additional arithmetic results available by changing the language is unwise, and that instead we should define appropriate functions in a package like math/bits. Here the best solution is a library change, not a language change.

A different problem we might have wanted to solve, after Go 1.0, was the fact that goroutines and shared memory make it too easy to introduce races into Go programs, causing crashes and other misbehavior in production. The language-based solution would have been to find some way to disallow data races, to make it impossible to write or at least to compile a program with a data race. How to fit that into a language like Go is still an open question in the programming language world. Instead we added a tool to the main distribution and made it trivial to use: that tool, the race detector, has become an indispensible part of the Go experience. Here the best solution was a runtime and tooling change, not a language change.

There will be language changes as well, of course, but not all problems are best solved in the language.

Shipping Go 2

Finally, how will we ship and deliver Go 2?

I think the best plan would be to ship the backwards-compatible parts of Go 2 incrementally, feature by feature, as part of the Go 1 release sequence. This has a few important properties. First, it keeps the Go 1 releases on the usual schedule, to continue the timely bug fixes and improvements that users now depend on. Second, it avoids splitting development effort between Go 1 and Go 2. Third, it avoids divergence between Go 1 and Go 2, to ease everyone's eventual migration. Fourth, it allows us to focus on and deliver one change at a time, which should help maintain quality. Fifth, it will encourage us to design features to be backwards-compatible.

We will need time to discuss and plan before any changes start landing in Go 1 releases, but it seems plausible to me that we might start seeing minor changes about a year from now, for Go 1.12 or so. That also gives us time to land package management support first.

Once all the backwards-compatible work is done, say in Go 1.20, then we can make the backwards-incompatible changes in Go 2.0. If there turn out to be no backwards-incompatible changes, maybe we just declare that Go 1.20 is Go 2.0. Either way, at that point we will transition from working on the Go 1.X release sequence to working on the Go 2.X sequence, perhaps with an extended support window for the final Go 1.X release.

This is all a bit speculative, and the specific release numbers I just mentioned are placeholders for ballpark estimates, but I want to make clear that we're not abandoning Go 1, and that in fact we will bring Go 1 along to the greatest extent possible.

Help Wanted

We need your help.

The conversation for Go 2 starts today, and it's one that will happen in the open, in public forums like the mailing list and the issue tracker. Please help us at every step along the way.

Today, what we need most is experience reports. Please tell us how Go is working for you, and more importantly not working for you. Write a blog post, include real examples, concrete detail, and real experience. And link it on our wiki page. That's how we'll start talking about what we, the Go community, might want to change about Go.

Thank you.

By Russ Cox

Introducing the Developer Experience Working Group

10 April 2017

Over the last several years, Go's audience has shifted from early adopters to mainstream users. Today, our users come from a wide variety of backgrounds, experiences, and expectations. The needs of users are growing faster than the Go project can currently address them. To streamline the experience for first-time Go users, we've created the Developer eXperience Working Group (DXWG).

For the next three months, this group will work together on delivering:

  • improvements to the Go installation experience
  • better guidelines to help new users
  • guides on tooling and developer environments (editors and IDEs)
  • running user studies to systematically analyze and measure friction points
  • improvements to the Go Tour and Go Playground

A secondary goal of the working group is to better understand how to involve the Go community in charting Go’s future. We hope that working groups – Go team members working alongside community members – will help Go scale its leadership and address user needs. We’ll learn from this experience and iterate.

The initial members of the working group are: Carmen Andoh, Chris Broadfoot, Francesc Campoy, Jaana Burcu Dogan, Steve Francia, Jess Frazelle, Bill Kennedy, Katrina Owen, Natalie Pistunovich, Mat Ryer, Dmitri Shuralyov.

We are looking for additional people to help with contributing code, writing documentation, sharing feedback and experiences (user stories), reviewing contributions, and more. If you are interested in any of our current areas of focus, please subscribe to the golang-devexp mailing list.

By The Developer Experience Working Group

HTTP/2 Server Push

24 March 2017

Introduction

HTTP/2 is designed to address many of the failings of HTTP/1.x. Modern web pages use many resources: HTML, stylesheets, scripts, images, and so on. In HTTP/1.x, each of these resources must be requested explicitly. This can be a slow process. The browser starts by fetching the HTML, then learns of more resources incrementally as it parses and evaluates the page. Since the server must wait for the browser to make each request, the network is often idle and underutilized.

To improve latency, HTTP/2 introduced server push, which allows the server to push resources to the browser before they are explicitly requested. A server often knows many of the additional resources a page will need and can start pushing those resources as it responds to the initial request. This allows the server to fully utilize an otherwise idle network and improve page load times.

At the protocol level, HTTP/2 server push is driven by PUSH_PROMISE frames. A PUSH_PROMISE describes a request that the server predicts the browser will make in the near future. As soon as the browser receives a PUSH_PROMISE, it knows that the server will deliver the resource. If the browser later discovers that it needs this resource, it will wait for the push to complete rather than sending a new request. This reduces the time the browser spends waiting on the network.

Server Push in net/http

Go 1.8 introduced support for pushing responses from an http.Server. This feature is available if the running server is an HTTP/2 server and the incoming connection uses HTTP/2. In any HTTP handler, you can assert if the http.ResponseWriter supports server push by checking if it implements the new http.Pusher interface.

For example, if the server knows that app.js will be required to render the page, the handler can initiate a push if http.Pusher is available:

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        if pusher, ok := w.(http.Pusher); ok {
            // Push is supported.
            if err := pusher.Push("/app.js", nil); err != nil {
                log.Printf("Failed to push: %v", err)
            }
        }
        // ...
    })

The Push call creates a synthetic request for /app.js, synthesizes that request into a PUSH_PROMISE frame, then forwards the synthetic request to the server's request handler, which will generate the pushed response. The second argument to Push specifies additional headers to include in the PUSH_PROMISE. For example, if the response to /app.js varies on Accept-Encoding, then the PUSH_PROMISE should include an Accept-Encoding value:

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        if pusher, ok := w.(http.Pusher); ok {
            // Push is supported.
            options := &http.PushOptions{
                Header: http.Header{
                    "Accept-Encoding": r.Header["Accept-Encoding"],
                },
            }
            if err := pusher.Push("/app.js", options); err != nil {
                log.Printf("Failed to push: %v", err)
            }
        }
        // ...
    })

A fully working example is available at:

$ go get golang.org/x/blog/content/h2push/server

If you run the server and load https://localhost:8080, your browser's developer tools should show that app.js and style.css were pushed by the server.

Start Your Pushes Before You Respond

It's a good idea to call the Push method before sending any bytes of the response. Otherwise it is possible to accidentally generate duplicate responses. For example, suppose you write part of an HTML response:

<html>
<head>
    <link rel="stylesheet" href="a.css">...

Then you call Push("a.css", nil). The browser may parse this fragment of HTML before it receives your PUSH_PROMISE, in which case the browser will send a request for a.css in addition to receiving your PUSH_PROMISE. Now the server will generate two responses for a.css. Calling Push before writing the response avoids this possibility entirely.

When To Use Server Push

Consider using server push any time your network link is idle. Just finished sending the HTML for your web app? Don't waste time waiting, start pushing the resources your client will need. Are you inlining resources into your HTML file to reduce latency? Instead of inlining, try pushing. Redirects are another good time to use push because there is almost always a wasted round trip while the client follows the redirect. There are many possible scenarios for using push -- we are only getting started.

We would be remiss if we did not mention a few caveats. First, you can only push resources your server is authoritative for -- this means you cannot push resources that are hosted on third-party servers or CDNs. Second, don't push resources unless you are confident they are actually needed by the client, otherwise your push wastes bandwidth. A corollary is to avoid pushing resources when it's likely that the client already has those resources cached. Third, the naive approach of pushing all resources on your page often makes performance worse. When in doubt, measure.

The following links make for good supplemental reading:

Conclusion

With Go 1.8, the standard library provides out-of-the-box support for HTTP/2 Server Push, giving you more flexibility to optimize your web applications.

Go to our HTTP/2 Server Push demo page to see it in action.

By Jaana Burcu Dogan, Tom Bergan

See the index for more articles.