12 July 2018
This is the transcript from the keynote I gave at the International Symposium on Memory Management (ISMM) on June 18, 2018. For the past 25 years ISMM has been the premier venue for publishing memory management and garbage collection papers and it was an honor to have been invited to give the keynote.
The Go language features, goals, and use cases have forced us to rethink the entire garbage collection stack and have led us to a surprising place. The journey has been exhilarating. This talk describes our journey. It is a journey motivated by open source and Google's production demands. Included are side hikes into dead end box canyons where numbers guided us home. This talk will provide insight into the how and the why of our journey, where we are in 2018, and Go's preparation for the next part of the journey.
Richard L. Hudson (Rick) is best known for his work in memory management including the invention of the Train, Sapphire, and Mississippi Delta algorithms as well as GC stack maps which enabled garbage collection in statically typed languages such as Modula-3, Java, C#, and Go. Rick is currently a member of Google's Go team where he is working on Go's garbage collection and runtime issues.
Comments: See the discussion on golang-dev.
Rick Hudson here.
This is a talk about the Go runtime and in particular the garbage collector. I have about 45 or 50 minutes of prepared material and after that we will have time for discussion and I'll be around so feel free to come up afterwards.
Before I get started I want to acknowledge some people.
A lot of the good stuff in the talk was done by Austin Clements. Other people on the Cambridge Go team, Russ, Than, Cherry, and David have been an engaging, exciting, and fun group to work with.
We also want to thank the 1.6 million Go users worldwide for giving us interesting problems to solve. Without them a lot of these problems would never come to light.
And finally I want to acknowledge Renee French for all these nice Gophers that she has been producing over the years. You will see several of them throughout the talk.
Before we get revved up and going on this stuff we really have to show what GC's view of Go looks like.
Well first of all Go programs have hundreds of thousands of stacks. They are managed by the Go scheduler and are always preempted at GC safepoints. The Go scheduler multiplexes Go routines onto OS threads which hopefully run with one OS thread per HW thread. We manage the stacks and their size by copying them and updating pointers in the stack. It's a local operation so it scales fairly well.
The next thing that is important is the fact that Go is a value-oriented language in the tradition of C-like systems languages rather than reference-oriented language in the tradition of most managed runtime languages. For example, this shows how a type from the tar package is laid out in memory. All of the fields are embedded directly in the Reader value. This gives programmers more control over memory layout when they need it. One can collocate fields that have related values which helps with cache locality.
Value-orientation also helps with the foreign function interfaces. We have a fast FFI with C and C++. Obviously Google has a tremendous number of facilities available but they are written in C++. Go couldn't wait to reimplement all of these things in Go so Go had to have access to these systems through the foreign function interface.
This one design decision has led to some of the more amazing things that have to go on with the runtime. It is probably the most important thing that differentiates Go from other GCed languages.
Of course Go can have pointers and in fact they can have interior pointers. Such pointers keep the entire value live and they are fairly common.
We also have a way ahead of time compilation system so the binary contains the entire runtime.
There is no JIT recompilation. There are pluses and minuses to this. First of all, reproducibility of program execution is a lot easier which makes moving forward with compiler improvements much faster.
On the sad side of it we don't have the chance to do feedback optimizations as you would with a JITed system.
So there are pluses and minuses.
Go comes with two knobs to control the GC. The first one is GCPercent. Basically this is a knob that adjusts how much CPU you want to use and how much memory you want to use. The default is 100 which means that half the heap is dedicated to live memory and half the heap is dedicated to allocation. You can modify this in either direction.
MaxHeap, which is not yet released but is being used and evaluated internally, lets the programmer set what the maximum heap size should be. Out of memory, OOMs, are tough on Go; temporary spikes in memory usage should be handled by increasing CPU costs, not by aborting. Basically if the GC sees memory pressure it informs the application that it should shed load. Once things are back to normal the GC informs the application that it can go back to its regular load. MaxHeap also provides a lot more flexibility in scheduling. Instead of always being paranoid about how much memory is available the runtime can size the heap up to the MaxHeap.
This wraps up our discussion on the pieces of Go that are important to the garbage collector.
So now let's talk about the Go runtime and how did we get here, how we got to where we are.
So it's 2014. If Go does not solve this GC latency problem somehow then Go isn't going to be successful. That was clear.
Other new languages were facing the same problem. Languages like Rust went a different way but we are going to talk about the path that Go took.
Why is latency so important?
The math is completely unforgiving on this.
A 99%ile isolated GC latency service level objective (SLO), such as 99% of the time a GC cycle takes < 10ms, just simply doesn't scale. What matters is latency during an entire session or through the course of using an app many times in a day. Assume a session that browses several web pages ends up making 100 server requests during a session or it makes 20 requests and you have 5 sessions packed up during the day. In that situation only 37% of users will have a consistent sub 10ms experience across the entire session.
If you want 99% of those users to have a sub 10ms experience, as we are suggesting, the math says you really need to target 4 9s or the 99.99%ile.
So it's 2014 and Jeff Dean had just come out with his paper called 'The Tail at Scale' which this digs into this further. It was being widely read around Google since it had serious ramifications for Google going forward and trying to scale at Google scale.
We call this problem the tyranny of the 9s.
So how do you fight the tyranny of the 9s?
A lot of things were being done in 2014.
If you want 10 answers ask for several more and take the first 10 and those are the answers you put on your search page. If the request exceeds 50%ile reissue or forward the request to another server. If GC is about to run, refuse new requests or forward the requests to another server until GC is done. And so forth and so on.
All these are workarounds come from very clever people with very real problems but they didn't tackle the root problem of GC latency. At Google scale we had to tackle the root problem. Why?
Redundancy wasn't going to scale, redundancy costs a lot. It costs new server farms.
We hoped we could solve this problem and saw it as an opportunity to improve the server ecosystem and in the process save some of the endangered corn fields and give some kernel of corn the chance to be knee high by the fourth of July and reach its full potential.
So here is the 2014 SLO. Yes, it was true that I was sandbagging, I was new on the team, it was a new process to me, and I didn't want to over promise.
Furthermore presentations about GC latency in other languages were just plain scary.
The original plan was to do a read barrier free concurrent copying GC. That was the long term plan. There was a lot of uncertainty about the overhead of read barriers so Go wanted to avoid them.
But short term 2014 we had to get our act together. We had to convert all of the runtime and compiler to Go. They were written in C at the time. No more C, no long tail of bugs due to C coders not understanding GC but having a cool idea about how to copy strings. We also needed something quickly and focused on latency but the performance hit had to be less than the speedups provided by the compiler. So we were limited. We had basically a year of compiler performance improvements that we could eat up by making the GC concurrent. But that was it. We couldn't slow down Go programs. That would have been untenable in 2014.
So we backed off a bit. We weren't going to do the copying part.
The decision was to do a tri-color concurrent algorithm. Earlier in my career Eliot Moss and I had done the journal proofs showing that Dijkstra's algorithm worked with multiple application threads. We also showed we could knock off the STW problems, and we had proofs that it could be done.
We were also concerned about compiler speed, that is the code the compiler generated. If we kept the write barrier turned off most of the time the compiler optimizations would be minimally impacted and the compiler team could move forward rapidly. Go also desperately needed short term success in 2015.
So let's look at some of the things we did.
We went with a size segregated span. Interior pointers were a problem.
The garbage collector needs to efficiently find the start of the object. If it knows the size of the objects in a span it simply rounds down to that size and that will be the start of the object.
Of course size segregated spans have some other advantages.
Low fragmentation: Experience with C, besides Google's TCMalloc and Hoard, I was intimately involved with Intel's Scalable Malloc and that work gave us confidence that fragmentation was not going to be a problem with non-moving allocators.
Internal structures: We fully understood and had experience with them. We understood how to do size segregated spans, we understood how to do low or zero contention allocation paths.
Speed: Non-copy did not concern us, allocation admittedly might be slower but still in the order of C. It might not be as fast as bump pointer but that was OK.
We also had this foreign function interface issue. If we didn't move our objects then we didn't have to deal with the long tail of bugs you might encounter if you had a moving collector as you attempt to pin objects and put levels of indirection between C and the Go object you are working with.
The next design choice was where to put the object's metadata. We needed to have some information about the objects since we didn't have headers. Mark bits are kept on the side and used for marking as well as allocation. Each word has 2 bits associated with it to tell you if it was a scalar or a pointer inside that word. It also encoded whether there were more pointers in the object so we could stop scanning objects sooner than later. We also had an extra bit encoding that we could use as an extra mark bit or to do other debugging things. This was really valuable for getting this stuff running and finding bugs.
So what about write barriers? The write barrier is on only during the GC. At other times the compiled code loads a global variable and looks at it. Since the GC was typically off the hardware correctly speculates to branch around the write barrier. When we are inside the GC that variable is different, and the write barrier is responsible for ensuring that no reachable objects get lost during the tri-color operations.
The other piece of this code is the GC Pacer. It is some of the great work that Austin did. It is basically based on a feedback loop that determines when to best start a GC cycle. If the system is in a steady state and not in a phase change, marking will end just about the time memory runs out.
That might not be the case so the Pacer also has to monitor the marking progress and ensure allocation doesn't overrun the concurrent marking.
If need be, the Pacer slows down allocation while speeding up marking. At a high level the Pacer stops the Goroutine, which is doing a lot of the allocation, and puts it to work doing marking. The amount of work is proportional to the Goroutine's allocation. This speeds up the garbage collector while slowing down the mutator.
When all of this is done the Pacer takes what it has learnt from this GC cycle as well as previous ones and projects when to start the next GC.
It does much more than this but that is the basic approach.
The math is absolutely fascinating, ping me for the design docs. If you are doing a concurrent GC you really owe it to yourself to look at this math and see if it's the same as your math. If you have any suggestions let us know.
Yes, so we had successes, lots of them. A younger crazier Rick would have taken some of these graphs and tattooed them on my shoulder I was so proud of them.
This is a series of graphs that was done for a production server at Twitter. We of course had nothing to do with that production server. Brian Hatfield did these measurements and oddly enough tweeted about them.
On the Y axis we have GC latency in milliseconds. On the X axis we have time. Each of the points is a stop the world pause time during that GC.
On our first release, which was in August of 2015, we saw a drop from around 300 - 400 milliseconds down to 30 or 40 milliseconds. This was good, order of magnitude good.
We are going to change the Y-axis here radically from 0 to 400 milliseconds down to 0 to 50 milliseconds.
This is 6 months later. The improvement was largely due to systematically eliminating all the O(heap) things we were doing during the stop the world time. This was our second order of magnitude improvement as we went from 40 milliseconds down to 4 or 5.
There were some bugs in there that we had to clean up and we did this during a minor release 1.6.3. This dropped latency down to well under 10 milliseconds, which was our SLO.
We are about to change our Y-axis again, this time down to 0 to 5 milliseconds.
So here we are, this is August of 2016, a year after the first release. Again we kept knocking off these O(heap size) stop the world processes. We are talking about an 18Gbyte heap here. We had much larger heaps and as we knocked off these O(heap size) stop the world pauses, the size of the heap could obviously grow considerable without impacting latency. So this was a bit of a help in 1.7.
The next release was in March of 2017. We had the last of our large latency drops which was due to figuring out how to avoid the stop the world stack scanning at the end of the GC cycle. That dropped us into the sub-millisecond range. Again the Y axis is about to change to 1.5 milliseconds and we see our third order of magnitude improvement.
The August 2017 release saw little improvement. We know what is causing the remaining pauses. The SLO whisper number here is around 100-200 microseconds and we will push towards that. If you see anything over a couple hundred microseconds then we really want to talk to you and figure out whether it fits into the stuff we know about or whether it is something new we haven't looked into. In any case there seems to be little call for lower latency. It is important to note these latency levels can happen for a wide variety of non-GC reasons and as the saying goes "You don't have to be faster than the bear, you just have to be faster than the guy next to you."
There was no substantial change in the Feb'18 1.10 release just some clean-up and chasing corner cases.
So a new year and a new SLO This is our 2018 SLO.
We have dropped total CPU to CPU used during a GC cycle.
The heap is still at 2x.
We now have an objective of 500 microseconds stop the world pause per GC cycle. Perhaps a little sandbagging here.
The allocation would continue to be proportional to the GC assists.
The Pacer had gotten much better so we looked to see minimal GC assists in a steady state.
We were pretty happy with this. Again this is not an SLA but an SLO so it's an objective, not an agreement, since we can't control such things as the OS.
That's the good stuff. Let's shift and start talking about our failures. These are our scars; they are sort of like tattoos and everyone gets them. Anyway they come with better stories so let's do some of those stories.
Our first attempt was to do something called the request oriented collector or ROC. The hypothesis can be seen here.
So what does this mean?
Goroutines are lightweight threads that look like Gophers, so here we have two Goroutines. They share some stuff such as the two blue objects there in the middle. They have their own private stacks and their own selection of private objects. Say the guy on the left wants to share the green object.
The goroutine puts it in the shared area so the other Goroutine can access it. They can hook it to something in the shared heap or assign it to a global variable and the other Goroutine can see it.
Finally the Goroutine on the left goes to its death bed, it's about to die, sad.
As you know you can't take your objects with you when you die. You can't take your stack either. The stack is actually empty at this time and the objects are unreachable so you can simply reclaim them.
The important thing here is that all actions were local and did not require any global synchronization. This is fundamentally different than approaches like a generational GC, and the hope was that the scaling we would get from not having to do that synchronization would be sufficient for us to have a win.
The other issue that was going on with this system was that the write barrier was always on. Whenever there was a write, we would have to see if it was writing a pointer to a private object into a public object. If so, we would have to make the referent object public and then do a transitive walk of reachable objects making sure they were also public. That was a pretty expensive write barrier that could cause many cache misses.
That said, wow, we had some pretty good successes.
This is an end-to-end RPC benchmark. The mislabeled Y axis goes from 0 to 5 milliseconds (lower is better), anyway that is just what it is. The X axis is basically the ballast or how big the in-core database is.
As you can see if you have ROC on and not a lot of sharing, things actually scale quite nicely. If you don't have ROC on it wasn't nearly as good.
But that wasn't good enough, we also had to make sure that ROC didn't slow down other pieces of the system. At that point there was a lot of concern about our compiler and we could not slow down our compilers. Unfortunately the compilers were exactly the programs that ROC did not do well at. We were seeing 30, 40, 50% and more slowdowns and that was unacceptable. Go is proud of how fast its compiler is so we couldn't slow the compiler down, certainly not this much.
We then went and looked at some other programs. These are our performance benchmarks. We have a corpus of 200 or 300 benchmarks and these were the ones the compiler folks had decided were important for them to work on and improve. These weren't selected by the GC folks at all. The numbers were uniformly bad and ROC wasn't going to become a winner.
It's true we scaled but we only had 4 to 12 hardware thread system so we couldn't overcome the write barrier tax. Perhaps in the future when we have 128 core systems and Go is taking advantage of them, the scaling properties of ROC might be a win. When that happens we might come back and revisit this, but for now ROC was a losing proposition.
So what were we going to do next? Let's try the generational GC. It's an oldie but a goodie. ROC didn't work so let's go back to stuff we have a lot more experience with.
We weren't going to give up our latency, we weren't going to give up the fact that we were non-moving. So we needed a non-moving generational GC.
So could we do this? Yes, but with a generational GC, the write barrier is always on. When the GC cycle is running we use the same write barrier we use today, but when GC is off we use a fast GC write barrier that buffers the pointers and then flushes the buffer to a card mark table when it overflows.
So how is this going to work in a non-moving situation? Here is the mark / allocation map. Basically you maintain a current pointer. When you are allocating you look for the next zero and when you find that zero you allocate an object in that space.
You then update the current pointer to the next 0.
You continue until at some point it is time to do a generation GC. You will notice that if there is a one in the mark/allocation vector then that object was alive at the last GC so it is mature. If it is zero and you reach it then you know it is young.
So how do you do promoting. If you find something marked with a 1 pointing to something marked with a 0 then you promote the referent simply by setting that zero to a one.
You have to do a transitive walk to make sure all reachable objects are promoted.
When all reachable objects have been promoted the minor GC terminates.
Finally, to finish your generational GC cycle you simply set the current pointer back to the start of the vector and you can continue. All the zeros weren't reached during that GC cycle so are free and can be reused. As many of you know this is called 'sticky bits' and was invented by Hans Boehm and his colleagues.
So what did the performance look like? It wasn't bad for the large heaps. These were the benchmarks that the GC should do well on. This was all good.
We then ran it on our performance benchmarks and things didn't go as well. So what was going on?
The write barrier was fast but it simply wasn't fast enough. Furthermore it was hard to optimize for. For example, write barrier elision can happen if there is an initializing write between when the object was allocated and the next safepoint. But we were having to move to a system where we have a GC safepoint at every instruction so there really wasn't any write barrier that we could elide going forward.
We also had escape analysis and it was getting better and better. Remember the value-oriented stuff we were talking about? Instead of passing a pointer to a function we would pass the actual value. Because we were passing a value, escape analysis would only have to do intraprocedural escape analysis and not interprocedural analysis.
Of course in the case where a pointer to the local object escapes, the object would be heap allocated.
It isn't that the generational hypothesis isn't true for Go, it's just that the young objects live and die young on the stack. The result is that generational collection is much less effective than you might find in other managed runtime languages.
So these forces against the write barrier were starting to gather. Today, our compiler is much better than it was in 2014. Escape analysis is picking up a lot of those objects and sticking them on the stack-objects that the generational collector would have helped with. We started creating tools to help our users find objects that escaped and if it was minor they could make changes to the code and help the compiler allocate on the stack.
Users are getting more clever about embracing value-oriented approaches and the number of pointers is being reduced. Arrays and maps hold values and not pointers to structs. Everything is good.
But that's not the main compelling reason why write barriers in Go have an uphill fight going forward.
Let's look at this graph. It's just an analytical graph of mark costs. Each line represents a different application that might have a mark cost. Say your mark cost is 20%, which is pretty high but it's possible. The red line is 10%, which is still high. The lower line is 5% which is about what a write barrier costs these days. So what happens if you double the heap size? That's the point on the right. The cumulative cost of the mark phase drops considerably since GC cycles are less frequent. The write barrier costs are constant so the cost of increasing the heap size will drive that marking cost underneath the cost of the write barrier.
Here is a more common cost for a write barrier, which is 4%, and we see that even with that we can drive the cost of the mark barrier down below the cost of the write barrier by simply increasing the heap size.
The real value of generational GC is that, when looking at GC times, the write barrier costs are ignored since they are smeared across the mutator. This is generational GC's great advantage, it greatly reduces the long STW times of full GC cycles but it doesn't necessarily improve throughput. Go doesn't have this stop the world problem so it had to look more closely at the throughput problems and that is what we did.
That's a lot of failure and with such failure comes food and lunch. I'm doing my usual whining "Gee wouldn't this be great if it wasn't for the write barrier."
Meanwhile Austin has just spent an hour talking to some of the HW GC folks at Google and he was saying we should talk to them and try and figure out how to get HW GC support that might help. Then I started telling war stories about zero-fill cache lines, restartable atomic sequences, and other things that didn't fly when I was working for a large hardware company. Sure we got some stuff into a chip called the Itanium, but we couldn't get them into the more popular chips of today. So the moral of the story is simply to use the HW we have.
Anyway that got us talking, what about something crazy?
What about card marking without a write barrier? It turns out that Austin has these files and he writes into these files all of his crazy ideas that for some reason he doesn't tell me about. I figure it is some sort of therapeutic thing. I used to do the same thing with Eliot. New ideas are easily smashed and one needs to protect them and make them stronger before you let them out into the world. Well anyway he pulls this idea out.
The idea is that you maintain a hash of mature pointers in each card. If pointers are written into a card, the hash will change and the card will be considered marked. This would trade the cost of write barrier off for cost of hashing.
But more importantly it's hardware aligned.
Today's modern architectures have AES (Advanced Encryption Standard) instructions. One of those instructions can do encryption-grade hashing and with encryption-grade hashing we don't have to worry about collisions if we also follow standard encryption policies. So hashing is not going to cost us much but we have to load up what we are going to hash. Fortunately we are walking through memory sequentially so we get really good memory and cache performance. If you have a DIMM and you hit sequential addresses, then it's a win because they will be faster than hitting random addresses. The hardware prefetchers will kick in and that will also help. Anyway we have 50 years, 60 years of designing hardware to run Fortran, to run C, and to run the SPECint benchmarks. It's no surprise that the result is hardware that runs this kind of stuff fast.
We took the measurement. This is pretty good. This is the benchmark suite for large heaps which should be good.
We then said what does it look like for the performance benchmark? Not so good, a couple of outliers. But now we have moved the write barrier from always being on in the mutator to running as part of the GC cycle. Now making a decision about whether we are going to do a generational GC is delayed until the start of the GC cycle. We have more control there since we have localized the card work. Now that we have the tools we can turn it over to the Pacer, and it could do a good job of dynamically cutting off programs that fall to the right and do not benefit from generational GC. But is this going to win going forward? We have to know or at least think about what hardware is going to look like going forward.
What are the memories of the future?
Let's take a look at this graph. This is your classic Moore's law graph. You have a log scale on the Y axis showing the number of transistors in a single chip. The X-axis is the years between 1971 and 2016. I will note that these are the years when someone somewhere predicted that Moore's law was dead.
Dennard scaling had ended frequency improvements ten years or so ago. New processes are taking longer to ramp. So instead of 2 years they are now 4 years or more. So it's pretty clear that we are entering an era of the slowing of Moore's law.
Let's just look at the chips in the red circle. These are the chips that are the best at sustaining Moore's law.
They are chips where the logic is increasingly simple and duplicated many times. Lots of identical cores, multiple memory controllers and caches, GPUs, TPUs, and so forth.
As we continue to simplify and increase duplication we asymptotically end up with a couple of wires, a transistor, and a capacitor. In other words a DRAM memory cell.
Put another way, we think that doubling memory is going to be a better value than doubling cores.
Original graph at www.kurzweilai.net/ask-ray-the-future-of-moores-law.
Let's look at another graph focused on DRAM. These are numbers from a recent PhD thesis from CMU. If we look at this we see that Moore's law is the blue line. The red line is capacity and it seems to be following Moore's law. Oddly enough I saw a graph that goes all the way back to 1939 when we were using drum memory and that capacity and Moore's law were chugging along together so this graph has been going on for a long time, certainly longer than probably anybody in this room has been alive.
If we compare this graph to CPU frequency or the various Moore's-law-is-dead graphs, we are led to the conclusion that memory, or at least chip capacity, will follow Moore's law longer than CPUs. Bandwidth, the yellow line, is related not only to the frequency of the memory but also to the number of pins one can get off of the chip so it's not keeping up as well but it's not doing badly.
Latency, the green line, is doing very poorly, though I will note that latency for sequential accesses does better than latency for random access.
(Data from "Understanding and Improving the Latency of DRAM-Based Memory Systems Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical and Computer Engineering Kevin K. Chang M.S., Electrical & Computer Engineering, Carnegie Mellon University B.S., Electrical & Computer Engineering, Carnegie Mellon University Carnegie Mellon University Pittsburgh, PA May, 2017". See Kevin K. Chang's thesis. The original graph in the introduction was not in a form that I could draw a Moore's law line on it easily so I changed the X-axis to be more uniform.)
Let's go to where the rubber meets the road. This is actual DRAM pricing and it has generally declined from 2005 to 2016. I chose 2005 since that is around the time when Dennard scaling ended and along with it frequency improvements.
If you look at the red circle, which is basically the time our work to reduce Go's GC latency has been going on, we see that for the first couple of years prices did well. Lately, not so good, as demand has exceeded supply leading to price increases over the last two years. Of course, transistors haven't gotten bigger and in some cases chip capacity has increased so this is driven by market forces. RAMBUS and other chip manufacturers say that moving forward we will see our next process shrink in the 2019-2020 time frame.
I will refrain from speculating on global market forces in the memory industry beyond noting that pricing is cyclic and in the long term supply has a tendency to meet demand.
Long term, it is our belief that memory pricing will drop at a rate that is much faster than CPU pricing.
Let's look at this other line. Gee it would be nice if we were on this line. This is the SSD line. It is doing a better job of keeping prices low. The material physics of these chips is much more complicated that with DRAM. The logic is more complex, instead of a one transistor per cell there are half a dozen or so.
Going forward there is a line between DRAM and SSD where NVRAM such as Intel's 3D XPoint and Phase Change Memory (PCM) will live. Over the next decade increased availability of this type of memory is likely to become more mainstream and this will only reinforce the idea that adding memory is the cheap way to add value to our servers.
More importantly we can expect to see other competing alternatives to DRAM. I won't pretend to know which one will be favored in five or ten years but the competition will be fierce and heap memory will move closer to the highlighted blue SSD line here.
All of this reinforces our decision to avoid always-on barriers in favor of increasing memory.
So what does all this mean for Go going forward?
We intend to make the runtime more flexible and robust as we look at corner cases that come in from our users. The hope is to tighten the scheduler down and get better determinism and fairness but we don't want to sacrifice any of our performance.
We also do not intend to increase the GC API surface. We've had almost a decade now and we have two knobs and that feels about right. There is not an application that is important enough for us to add a new flag.
We will also be looking into how to improve our already pretty good escape analysis and optimize for Go's value-oriented programming. Not only in the programming but in the tools we provide our users.
Algorithmically, we will focus on parts of the design space that minimize the use of barriers, particularly those that are turned on all the time.
Finally, and most importantly, we hope to ride Moore's law's tendency to favor RAM over CPU certainly for the next 5 years and hopefully for the next decade.
So that's it. Thank you.
P.S. The Go team is looking to hire engineers to help develop and maintain the Go runtime and compiler toolchain.
Interested? Have a look at our open positions.
23 May 2018
In November 2015, we introduced the Go Code of Conduct. It was developed in a collaboration between the Go team members at Google and the Go community. I was fortunate to be one of the community members invited to participate in both drafting and then enforcing the Go Code of Conduct. Since then, we have learned two lessons about limitations in our code of conduct that restricted us from being able to cultivate the safe culture essential to Go’s success.
The first lesson we learned is that toxic behaviors by project participants in non-project spaces can have a negative impact on the project affecting the security and safety of community members. There were a few reported incidents where actions took place outside of project spaces but the impact was felt inside our community. The specific language in our code of conduct restricted our ability to respond only to actions happening “in the official forums operated by the Go project”. We needed a way to protect our community members wherever they are.
The second lesson we learned is that the demands required to enforce the code of conduct place too heavy of a burden on volunteers. The initial version of the code of conduct presented the working group as disciplinarians. It was soon clear that this was too much, so in early 2017 we changed the group’s role to that of advisors and mediators. Still, working group community members reported feeling overwhelmed, untrained, and vulnerable. This well-intentioned shift left us without an enforcement mechanism without solving the issue with overburdened volunteers.
In mid-2017, I represented the Go project in a meeting with Google’s Open Source Programs Office and Open Source Strategy Team to address the shortcomings in our respective codes of conduct, particularly in their enforcement. It quickly became clear that our problems had a lot in common, and that working together on a single code of conduct for all of Google’s open source projects made sense. We started with the text from the Contributor Covenant Code of Conduct v1.4 and then made changes, influenced by our experiences in Go community and our collective experiences in open source. This resulted in the Google code of conduct template.
Today the Go project is adopting this new code of conduct, and we’ve updated golang.org/conduct. This revised code of conduct retains much of the intent, structure and language of the original Go code of conduct while making two critical changes that address the shortcomings identified above.
First, the new code of conduct makes clear that people who participate in any kind of harassment or inappropriate behavior, even outside our project spaces, are not welcome in our project spaces. This means that the Code of Conduct applies outside the project spaces when there is a reasonable belief that an individual’s behavior may have a negative impact on the project or its community.
Second, in the place of the working group, the new code of conduct introduces a single Project Steward who will have explicit training and support for this role. The Project Steward will receive reported violations and then work with a committee, consisting of representatives from the Open Source Programs Office and the Google Open Source Strategy team, to find a resolution.
Our first Project Steward will be Cassandra Salisbury. She is well known to the Go community as a member of Go Bridge, an organizer of many Go meetups and conferences, and as a lead of the Go community outreach working group. Cassandra now works on the Go team at Google with a focus on advocating for and supporting the Go community.
We are grateful to everyone who served on the original Code of Conduct Working Group. Your efforts were essential in creating an inclusive and safe community.
We believe the code of conduct has contributed to the Go project becoming more welcoming now than it was in 2015, and we should all be proud of that.
We hope that the new code of conduct will help protect our community members even more effectively.
26 April 2018
I am delighted to announce the launch of Go’s new look and logo.
Go has been on an amazing journey over the last 8+ years. Our project, community and users have evolved through this journey and we wanted the Go brand to reflect where we have been and convey where we are going.
Go’s new brand is about our identity, our values, and our users. Over the past several months we have worked with a brand agency to develop a brand guide for Go. Our agency, Within, coordinated with and built upon the great foundation that Renee French established. Rest easy, our beloved Gopher Mascot remains at the center of our brand.
Our logo follows the brand’s core philosophy of simplicity over complexity. Using a modern, italicized sans-serif typeface combined with three simple motion lines forms a mark that resembles two wheels in rapid motion, communicating speed and efficiency. The circular shape of the letters hints at the eyes of the Go gopher, creating a familiar shape and allowing the mark and the mascot to pair well together.
Our new logo went through an extensive design process. Here are some of the revisions we went through…
New Brand Guide
The brand guide establishes the mission, values and voice for the Go project. In general terms, a brand guide is a reference tool for establishing a consistent brand identity. It is sometimes referred to as “brand guidelines” or a “brand book”. It is a single document that serves as a guide and reference to designers, writers, and developers to create consistent, on-brand content.
New presentation themes
In addition to our brand guide we have also developed a presentation theme. This presentation theme will enable us to have a consistent representation of Go in person at meetups and conferences as well as online. Go community members are welcome to use this theme for their own presentations.
The presentations are available as Google Slides presentations. We chose Google slides as it is easy to share and maintain updates. People are welcome to port them to keynote, powerpoint, etc.
Like this blog and all our gopher images, the slide themes are Creative Commons Attribution 3.0 licensed. The photos in the slides are all from unsplash and are released under the unsplash license.
Instructions to use slides:
- Open the Go Slide Masters presentation on Google slides.
- File > “Make a Copy” (you may need to login first)
- Create new slides using the layouts provided in the layouts menu.
- Use the included example slides to help guide the styling and creation of your presentation.
The brand guide, logo and themes are copyrighted by the Go authors.
The brand guide contains the guidelines for acceptable logo use.
What’s happening next
The website will be getting a refresh based on the new design. Since we are making significant changes, we are also taking this opportunity to update our website infrastructure to better serve our global community with internationalization and multilingual support. The migration will happen in stages over the next few months starting with this blog.
26 March 2018
Eight years ago, the Go team introduced
(which led to
and with it the decentralized, URL-like import paths
that Go developers are familiar with today.
After we released
goinstall, one of the first questions people asked
was how to incorporate version information.
We admitted we didn’t know.
For a long time, we believed that the problem of package versioning
would be best solved by an add-on tool,
and we encouraged people to create one.
The Go community created many tools with different approaches.
Each one helped us all better understand the problem,
but by mid-2016 it was clear that there were now too many solutions.
We needed to adopt a single, official tool.
After a community discussion started at GopherCon in July 2016 and continuing into the fall,
we all believed the answer would be to follow the package versioning approach
exemplified by Rust’s Cargo, with tagged semantic versions,
a manifest, a lock file, and a
SAT solver to decide which versions to use.
Sam Boyer led a team to create Dep, which followed this rough plan,
and which we intended to serve as the model for
go command integration.
But as we learned more about the implications of the Cargo/Dep approach,
it became clear to me that Go would benefit from changing
some of the details, especially concerning backwards compatibility.
The Impact of Compatibility
The most important new feature of Go 1 was not a language feature. It was Go 1’s emphasis on backwards compatibility. Until that point we’d issued stable release snapshots approximately monthly, each with significant incompatible changes. We observed significant acceleration in interest and adoption immediately after the release of Go 1. We believe that the promise of compatibility made developers feel much more comfortable relying on Go for production use and is a key reason that Go is popular today. Since 2013 the Go FAQ has encouraged package developers to provide their own users with similar expectations of compatibility. We call this the import compatibility rule: “If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.”
Independently, semantic versioning has become the de facto standard for describing software versions in many language communities, including the Go community. Using semantic versioning, later versions are expected to be backwards-compatible with earlier versions, but only within a single major version: v1.2.3 must be compatible with v1.2.1 and v1.1.5, but v2.3.4 need not be compatible with any of those.
If we adopt semantic versioning for Go packages,
as most Go developers expect,
then the import compatibility rule requires that
different major versions must use different import paths.
This observation led us to semantic import versioning,
in which versions starting at v2.0.0 include the major
version in the import path:
A year ago I strongly believed that whether to include version numbers in import paths was largely a matter of taste, and I was skeptical that having them was particularly elegant. But the decision turns out to be a matter not of taste but of logic: import compatibility and semantic versioning together require semantic import versioning. When I realized this, the logical necessity surprised me.
I was also surprised to realize that there is a second, independent logical route to semantic import versioning: gradual code repair or partial code upgrades. In a large program, it’s unrealistic to expect all packages in the program to update from v1 to v2 of a particular dependency at the same time. Instead, it must be possible for some of the program to keep using v1 while other parts have upgraded to v2. But then the program’s build, and the program’s final binary, must include both v1 and v2 of the dependency. Giving them the same import path would lead to confusion, violating what we might call the import uniqueness rule: different packages must have different import paths. The only way to have partial code upgrades, import uniqueness, and semantic versioning is to adopt semantic import versioning as well.
It is of course possible to build systems that use semantic versioning without semantic import versioning, but only by giving up either partial code upgrades or import uniqueness. Cargo allows partial code upgrades by giving up import uniqueness: a given import path can have different meanings in different parts of a large build. Dep ensures import uniqueness by giving up partial code upgrades: all packages involved in a large build must find a single agreed-upon version of a given dependency, raising the possibility that large programs will be unbuildable. Cargo is right to insist on partial code upgrades, which are critical to large-scale software development. Dep is equally right to insist on import uniqueness. Complex uses of Go’s current vendoring support can violate import uniqueness. When they have, the resulting problems have been quite challenging for both developers and tools to understand. Deciding between partial code upgrades and import uniqueness requires predicting which will hurt more to give up. Semantic import versioning lets us avoid the choice and keep both instead.
I was also surprised to discover how much import compatibility simplifies version selection, which is the problem of deciding which package versions to use for a given build. The constraints of Cargo and Dep make version selection equivalent to solving Boolean satisfiability, meaning it can be very expensive to determine whether a valid version configuration even exists. And then there may be many valid configurations, with no clear criteria for choosing the “best” one. Relying on import compatibility can instead let Go use a trivial, linear-time algorithm to find the single best configuration, which always exists. This algorithm, which I call minimal version selection, in turn eliminates the need for separate lock and manifest files. It replaces them with a single, short configuration file, edited directly by both developers and tools, that still supports reproducible builds.
Our experience with Dep demonstrates the impact of compatibility. Following the lead of Cargo and earlier systems, we designed Dep to give up import compatibility as part of adopting semantic versioning. I don’t believe we decided this deliberately; we just followed those other systems. The first-hand experience of using Dep helped us better understand exactly how much complexity is created by permitting incompatible import paths. Reviving the import compatibility rule by introducing semantic import versioning eliminates that complexity, leading to a much simpler system.
Progress, a Prototype, and a Proposal
Dep was released in January 2017. Its basic model—code tagged with semantic versions, along with a configuration file that specified dependency requirements—was a clear step forward from most of the Go vendoring tools, and converging on Dep itself was also a clear step forward. I wholeheartedly encouraged its adoption, especially to help developers get used to thinking about Go package versions, both for their own code and their dependencies. While Dep was clearly moving us in the right direction, I had lingering concerns about the complexity devil in the details. I was particularly concerned about Dep lacking support for gradual code upgrades in large programs. Over the course of 2017, I talked to many people, including Sam Boyer and the rest of the package management working group, but none of us could see any clear way to reduce the complexity. (I did find many approaches that added to it.) Approaching the end of the year, it still seemed like SAT solvers and unsatisfiable builds might be the best we could do.
In mid-November, trying once again to work through how Dep could support gradual code upgrades, I realized that our old advice about import compatibility implied semantic import versioning. That seemed like a real breakthrough. I wrote a first draft of what became my semantic import versioning blog post, concluding it by suggesting that Dep adopt the convention. I sent the draft to the people I’d been talking to, and it elicited very strong responses: everyone loved it or hated it. I realized that I needed to work out more of the implications of semantic import versioning before circulating the idea further, and I set out to do that.
In mid-December, I discovered that import compatibility and semantic import versioning together allowed cutting version selection down to minimal version selection. I wrote a basic implementation to be sure I understood it, I spent a while learning the theory behind why it was so simple, and I wrote a draft of the post describing it. Even so, I still wasn’t sure the approach would be practical in a real tool like Dep. It was clear that a prototype was needed.
In January, I started work on a simple
go command wrapper
that implemented semantic import versioning
and minimal version selection.
Trivial tests worked well.
Approaching the end of the month,
my simple wrapper could build Dep,
a real program that made use of many versioned packages.
The wrapper still had no command-line interface—the fact that
it was building Dep was hard-coded in a few string constants—but
the approach was clearly viable.
I spent the first three weeks of February turning the
wrapper into a full versioned
writing drafts of a
blog post series introducing
and discussing them with
Sam Boyer, the package management working group,
and the Go team.
And then I spent the last week of February finally
vgo and the ideas behind it with the whole Go community.
In addition to the core ideas of import compatibility,
semantic import versioning, and minimal version selection,
vgo prototype introduces a number of smaller
but significant changes motivated by eight years of
the new concept of a Go module,
which is a collection of packages versioned as a unit;
verifiable and verified builds;
version-awareness throughout the
enabling work outside
and the elimination of (most)
The result of all of this is the official Go proposal,
which I filed last week.
Even though it might look like a complete implementation,
it’s still just a prototype,
one that we will all need to work together to complete.
You can download and try the
vgo prototype from golang.org/x/vgo,
and you can read the
Tour of Versioned Go
to get a sense of what using
vgo is like.
The Path Forward
The proposal I filed last week is exactly that: an initial proposal. I know there are problems with it that the Go team and I can’t see, because Go developers use Go in many clever ways that we don’t know about. The goal of the proposal feedback process is for us all to work together to identify and address the problems in the current proposal, to make sure that the final implementation that ships in a future Go release works well for as many developers as possible. Please point out problems on the proposal discussion issue. I will keep the discussion summary and FAQ updated as feedback arrives.
For this proposal to succeed, the Go ecosystem as a whole—and in particular today’s major Go projects—will need to adopt the import compatibility rule and semantic import versioning. To make sure that can happen smoothly, we will also be conducting user feedback sessions by video conference with projects that have questions about how to incorporate the new versioning proposal into their code bases or have feedback about their experiences. If you are interested in participating in such a session, please email Steve Francia at email@example.com.
We’re looking forward to (finally!) providing the Go community with a single, official answer
to the question of how to incorporate package versioning into
Thanks to everyone who helped us get this far, and to everyone who will help us going forward.
We hope that, with your help, we can ship something that Go developers will love.
26 February 2018
This post summarizes the result of our 2017 user survey along with commentary and insights. It also draws key comparisons between the results of the 2016 and 2017 survey.
This year we had 6,173 survey respondents, 70% more than the 3,595 we had in the Go 2016 User Survey. In addition, it also had a slightly higher completion rate (84% → 87%) and a higher response rate to most of the questions. We believe that survey length is the main cause of this improvement as the 2017 survey was shortened in response to feedback that the 2016 survey was too long.
We are grateful to everyone who provided their feedback through the survey to help shape the future of Go.
For the first time, more survey respondents say they are paid to write Go than say they write it outside work. This indicates a significant shift in Go's user base and in its acceptance by companies for professional software development.
The areas people who responded to the survey work in is mostly consistent with last year, however, mobile and desktop applications have fallen significantly.
Another important shift: the #1 use of Go is now writing API/RPC services (65%, up 5% over 2016), taking over the top spot from writing CLI tools in Go (63%). Both take full advantage of Go's distinguishing features and are key elements of modern cloud computing. As more companies adopt Go, we expect these two uses of Go to continue to thrive.
Most of the metrics reaffirm things we have learned in prior years. Go programmers still overwhelmingly prefer Go. As more time passes Go users are deepening their experience in Go. While Go has increased its lead among Go developers, the order of language rankings remains quite consistent with last year.
In nearly every question around the usage and perception of Go, Go has demonstrated improvement over our prior survey. Users are happier using Go, and a greater percentage prefer using Go for their next project.
When asked about the biggest challenges to their own personal use of Go, users clearly conveyed that lack of dependency management and lack of generics were their two biggest issues, consistent with 2016. In 2017 we laid a foundation to be able to address these issues. We improved our proposal and development process with the addition of Experience Reports which is enabling the project to gather and obtain feedback critical to making these significant changes. We also made sigificant changes under the hood in how Go obtains, and builds packages. This is foundational work essential to addressing our dependency management needs.
These two issues will continue to be a major focus of the project through 2018.
In this section we asked two new questions. Both center around what developers are doing with Go in a more granular way than we've previously asked. We hope this data will provide insights for the Go project and ecosystem.
Since last year there has been an increase of the percentage of people who identified "Go lacks critical features" as the reason they don't use Go more and a decreased percentage who identified "Go not being an appropriate fit". Other than these changes, the list remains consistent with last year.
Reading the data: This question asked how strongly the respondent agreed or disagreed with the statement. The responses for each statement are displayed as sections of a single bar, from “strongly disagree” in deep red on the left end to “strongly agree” in deep blue on the right end. The bars use the same scale as the rest of the graphs, so they can (and do, especially later in the survey) vary in overall length due to lack of responses.
The ratio after the text compares the number of respondents who agreed (including “somewhat agree” and “strongly agree”) to those who disagreed (including “somewhat disagree” and “strongly disagree”). For example, the ratio of respondents agreeing that they would recommend Go to respondents disagreeing was 19 to 1. The second ratio (within the brackets) is simply a weighted ratio with each somewhat = 1, agree/disagree = 2, and strongly = 4.
Reading the data: This question asked for write-in responses. The bars above show the fraction of surveys mentioning common words or phrases. Only words or phrases that appeared in 20 or more surveys are listed, and meaningless common words or phrases like “the” or “to be” are omitted. The displayed results do overlap: for example, the 402 responses that mentioned “management” do include the 266 listed separately that mentioned “dependency management” and the 79 listed separately that mentioned “package management.” However, nearly or completely redundant shorter entries are omitted: there are not twenty or more surveys that listed “dependency” without mentioning “dependency management,” so there is no separate entry for “dependency.”
Development and deployment
We asked programmers which operating systems they develop Go on; the ratios of their responses remain consistent with last year. 64% of respondents say they use Linux, 49% use MacOS, and 18% use Windows, with multiple choices allowed.
Continuing its explosive growth, VSCode is now the most popular editor among Gophers. IntelliJ/GoLand also saw significant increase in usage. These largely came at the expense of Atom and Submlime Text which saw relative usage drops. This question had a 6% higher response rate from last year.
Survey respondents demonstrated significantly higher satisfaction with Go support in their editors over 2016 with the ratio of satisfied to dissatisfied doubling (9:1 → 18:1). Thank you to everyone who worked on Go editor support for all your hard work.
Go deployment is roughly evenly split between privately managed servers and hosted cloud servers. For Go applications, Google Cloud services saw significant increase over 2016. For Non-Go applications, AWS Lambda saw the largest increase in use.
We asked how strongly people agreed or disagreed with various statements about Go. All questions are repeated from last year with the addition of one new question which we introduced to add further clarifaction around how users are able to both find and use Go libraries.
All responses either indicated a small improvement or are comparable to 2016.
As in 2016, the most commonly requested missing library for Go is one for writing GUIs though the demand is not as pronounced as last year. No other missing library registered a significant number of responses.
The primary sources for finding answers to Go questions are the Go web site, Stack Overflow, and reading source code directly. Stack Overflow showed a small increase from usage over last year.
The primary sources for Go news are still the Go blog, Reddit’s /r/golang and Twitter; like last year, there may be some bias here since these are also how the survey was announced.
The Go Project
59% of respondents expressed interest in contributing in some way to the Go community and projects, up from 55% last year. Respondents also indicated that they felt much more welcome to contribute than in 2016. Unfortunately, respondents indicated only a very tiny improvement in understanding how to contribute. We will be actively working with the community and its leaders to make this a more accessible process.
Respondents showed an increase in agreement that they are confident in the leadership of the Go project (9:1 → 11:1). They also showed a small increase in agreement that the project leadership understands their needs (2.6:1 → 2.8:1) and in agreement that they feel comfortable approaching project leadership with questions and feedback (2.2:1 → 2.4:1). While improvements were made, this continues to be an area of focus for the project and its leadership going forward. We will continue to work to improve our understanding of user needs and approachability.
We tried some new ways to engage with users in 2017 and while progress was made, we are still working on making these solutions scalable for our growing community.
At the end of the survey, we asked some demographic questions.
The country distribution of responses is largely similar to last year with minor fluctuations. Like last year, the distribution of countries is similar to the visits to golang.org, though some Asian countries remain under-represented in the survey.
Perhaps the most significant improvement over 2016 came from the question which asked to what degree do respondents agreed with the statement, "I feel welcome in the Go community". Last year the agreement to disagreement ratio was 15:1. In 2017 this ratio nearly doubled to 25:1.
An important part of a community is making everyone feel welcome, especially people from under-represented demographics. We asked an optional question about identification across a few underrepresented groups. We had a 4% increase in response rate over last year. The percentage of each underrepresented group increased over 2016, some quite significantly.
Like last year, we took the results of the statement “I feel welcome in the Go community” and broke them down by responses to the various underrepresented categories. Like the whole, most of the respondents who identified as underrepresented also felt significantly more welcome in the Go community than in 2016. Respondents who identified as a woman showed the most significant improvement with an increase of over 400% in the ratio of agree:disagree to this statement (3:1 → 13:1). People who identified as ethnically or racially underrepresented had an increase of over 250% (7:1 → 18:1). Like last year, those who identified as not underrepresented still had a much higher percentage of agreement to this statement than those identifying from underrepresented groups.
We are encouraged by this progress and hope that the momentum continues.
The final question on the survey was just for fun: what’s your favorite Go
keyword? Perhaps unsurprisingly, the most popular response was
go, followed by
select, unchanged from last year.
Finally, on behalf of the entire Go project, we are grateful for everyone who has contributed to our project, whether by being a part of our great community, by taking this survey or by taking an interest in Go.
See the index for more articles.