10 November 2015
Six years ago today the Go language was released as an open source project. Since then, more than 780 contributors have made over 30,000 commits to the project's 22 repositories. The ecosystem continues to grow, with GitHub reporting more than 90,000 Go repositories. And, offline, we see new Go events and user groups pop up around the world with regularity.
In August we released Go 1.5, the most significant release since Go 1. It features a completely redesigned garbage collector that makes the language more suitable for latency-sensitive applications; it marks the transition from a C-based compiler tool chain to one written entirely in Go; and it includes ports to new architectures, with better support for ARM processors (the chips that power most smartphones). These improvements make Go better suited to a broader range of tasks, a trend that we hope will continue over the coming years.
Improvements to tools continue to boost developer productivity. We introduced the execution tracer and the "go doc" command, as well as more enhancements to our various static analysis tools. We are also working on an official Go plugin for Sublime Text, with better support for other editors in the pipeline.
Early next year we will release more improvements in Go 1.6, including HTTP/2 support for net/http servers and clients, an official package vendoring mechanism, support for blocks in text and HTML templates, a memory sanitizer that checks both Go and C/C++ code, and the usual assortment of other improvements and fixes.
This is the sixth time we have had the pleasure of writing a birthday blog post for Go, and we would not be doing so if not for the wonderful and passionate people in our community. The Go team would like to thank everyone who has contributed code, written an open source library, authored a blog post, helped a new gopher, or just given Go a try. Without you, Go would not be as complete, useful, or successful as it is today. Thank you, and celebrate!
9 October 2015
On August 21st the Go community gathered in London for the first edition of Golang UK. The conference featured two parallel tracks and nearly 400 gophers attended.
- Stupid Gopher Tricks, by Andrew Gerrand (video)
- Complex Concurrency Patterns in Go, by Evan Huus (video)
- Code Analysis [no reading required], by Francesc Campoy (video)
- Go kit: a toolkit for microservices Peter Bourgon (video)
- Dependency Management Conundrum, by William Kennedy (video)
- Building APIs, by Mat Ryer (video)
- Building a Bank with Go, by Matt Heath (video)
- CockroachDB: Make Data Easy, by Ben Darnell (video)
- Understanding memory allocation in Go, by Dean Elbaz (video)
- Whispered Secrets, by Eleanor McHugh (video)
On the day before the conference William Kennedy gave a full day Go workshop.
It was a great conference, so congratulations to the organizers and see you next year in London!
31 August 2015
Go is building a garbage collector (GC) not only for 2015 but for 2025 and beyond: A GC that supports today’s software development and scales along with new software and hardware throughout the next decade. Such a future has no place for stop-the-world GC pauses, which have been an impediment to broader uses of safe and secure languages such as Go.
Go 1.5, the first glimpse of this future, achieves GC latencies well below the 10 millisecond goal we set a year ago. We presented some impressive numbers in a talk at Gophercon. The latency improvements have generated a lot of attention; Robin Verlangen’s blog post Billions of requests per day meet Go 1.5 validates our direction with end to end results. We also particularly enjoyed Alan Shreve’s production server graphs and his "Holy 85% reduction" comment.
Today 16 gigabytes of RAM costs $100 and CPUs come with many cores, each with multiple hardware threads. In a decade this hardware will seem quaint but the software being built in Go today will need to scale to meet expanding needs and the next big thing. Given that hardware will provide the power to increase throughput, Go’s garbage collector is being designed to favor low latency and tuning via only a single knob. Go 1.5 is the first big step down this path and these first steps will forever influence Go and the applications it best supports. This blog post gives a high-level overview of what we have done for the Go 1.5 collector.
To create a garbage collector for the next decade, we turned to an algorithm from decades ago. Go's new garbage collector is a concurrent, tri-color, mark-sweep collector, an idea first proposed by Dijkstra in 1978. This is a deliberate divergence from most "enterprise" grade garbage collectors of today, and one that we believe is well suited to the properties of modern hardware and the latency requirements of modern software.
In a tri-color collector, every object is either white, grey, or black and we view the heap as a graph of connected objects. At the start of a GC cycle all objects are white. The GC visits all roots, which are objects directly accessible by the application such as globals and things on the stack, and colors these grey. The GC then chooses a grey object, blackens it, and then scans it for pointers to other objects. When this scan finds a pointer to a white object, it turns that object grey. This process repeats until there are no more grey objects. At this point, white objects are known to be unreachable and can be reused.
This all happens concurrently with the application, known as the mutator, changing pointers while the collector is running. Hence, the mutator must maintain the invariant that no black object points to a white object, lest the garbage collector lose track of an object installed in a part of the heap it has already visited. Maintaining this invariant is the job of the write barrier, which is a small function run by the mutator whenever a pointer in the heap is modified. Go’s write barrier colors the now-reachable object grey if it is currently white, ensuring that the garbage collector will eventually scan it for pointers.
Deciding when the job of finding all grey objects is done is subtle and can be expensive and complicated if we want to avoid blocking the mutators. To keep things simple Go 1.5 does as much work as it can concurrently and then briefly stops the world to inspect all potential sources of grey objects. Finding the sweet spot between the time needed for this final stop-the-world and the total amount of work that this GC does is a major deliverable for Go 1.6.
Of course the devil is in the details. When do we start a GC cycle? What metrics do we use to make that decision? How should the GC interact with the Go scheduler? How do we pause a mutator thread long enough to scan its stack? How do we represent white, grey, and black so we can efficiently find and scan grey objects? How do we know where the roots are? How do we know where in an object pointers are located? How do we minimize memory fragmentation? How do we deal with cache performance issues? How big should the heap be? And on and on, some related to allocation, some to finding reachable objects, some related to scheduling, but many related to performance. Low-level discussions of each of these areas are beyond the scope of this blog post.
At a higher level, one approach to solving performance problems is to add GC knobs, one for each performance issue. The programmer can then turn the knobs in search of appropriate settings for their application. The downside is that after a decade with one or two new knobs each year you end up with the GC Knobs Turner Employment Act. Go is not going down that path. Instead we provide a single knob, called GOGC. This value controls the total size of the heap relative to the size of reachable objects. The default value of 100 means that total heap size is now 100% bigger than (i.e., twice) the size of the reachable objects after the last collection. 200 means total heap size is 200% bigger than (i.e., three times) the size of the reachable objects. If you want to lower the total time spent in GC, increase GOGC. If you want to trade more GC time for less memory, lower GOGC.
More importantly as RAM doubles with the next generation of hardware, simply doubling GOGC will halve the number of GC cycles. On the other hand since GOGC is based on reachable object size, doubling the load by doubling the reachable objects requires no retuning. The application just scales. Furthermore, unencumbered by ongoing support for dozens of knobs, the runtime team can focus on improving the runtime based on feedback from real customer applications.
Go 1.5’s GC ushers in a future where stop-the-world pauses are no longer a barrier to moving to a safe and secure language. It is a future where applications scale effortlessly along with hardware and as hardware becomes more powerful the GC will not be an impediment to better, more scalable software. It’s a good place to be for the next decade and beyond. For more details about the 1.5 GC and how we eliminated latency issues see the Go GC: Latency Problem Solved presentation or the slides.
19 August 2015
Today the Go project is proud to release Go 1.5, the sixth major stable release of Go.
This release includes significant changes to the implementation. The compiler tool chain was translated from C to Go, removing the last vestiges of C code from the Go code base. The garbage collector was completely redesigned, yielding a dramatic reduction in garbage collection pause times. Related improvements to the scheduler allowed us to change the default GOMAXPROCS value (the number of concurrently executing goroutines) from 1 to the number of logical CPUs. Changes to the linker enable distributing Go packages as shared libraries to link into Go programs, and building Go packages into archives or shared libraries that may be linked into or loaded by C programs (design doc).
The release also includes improvements to the developer tools. Support for "internal" packages permits sharing implementation details between packages. Experimental support for "vendoring" external dependencies is a step toward a standard mechanism for managing dependencies in Go programs. The new "go tool trace" command enables the visualisation of program traces generated by new tracing infrastructure in the runtime. The new "go doc" command provides an improved command-line interface for viewing Go package documentation.
There are also several new operating system and architecture ports. The more mature new ports are darwin/arm, darwin/arm64 (Apple's iPhone and iPad devices), and linux/arm64. There is also experimental support for ppc64 and ppc64le (IBM 64-bit PowerPC, big and little endian).
The new darwin/arm64 port and external linking features fuel the Go mobile project, an experiment to see how Go might be used for building apps on Android and iOS devices. (The Go mobile work itself is not part of this release.)
The only language change is very minor, the lifting of a restriction in the map literal syntax to make them more succinct and consistent with slice literals.
The standard library saw many additions and improvements, too. The flag package now shows cleaner usage messages. The math/big package now provides a Float type for computing with arbitrary-precision floating point numbers. An improvement to the DNS resolver on Linux and BSD systems has removed the cgo requirement for programs that do name lookups. The go/types package has been moved to the standard library from the golang.org/x/tools repository. (The new go/constant and go/importer packages are also a result of this move.) The reflect package has added the ArrayOf and FuncOf functions, analogous to the existing SliceOf function. And, of course, there is the usual list of smaller fixes and improvements.
28 July 2015
A few weeks ago, Go programmers from around the world descended on Denver, Colorado for GopherCon 2015. The two-day, single-track conference attracted more than 1,250 attendees—nearly double last year's number—and featured 22 talks presented by Go community members.
The Cowboy Gopher (a toy given to each attendee) watches over the ranch.
Photograph by Nathan Youngman. Gopher by Renee French.
Today the organizers have posted the videos online so you can now enjoy the conference from afar:
- Go, Open Source, Community — Russ Cox (video) (text)
- Go kit: A Standard Library for Distributed Programming — Peter Bourgon (video) (slides)
- Delve Into Go — Derek Parker (video) (slides)
- How a complete beginner learned Go as her first backend language in 5 weeks — Audrey Lim (video) (slides)
- A Practical Guide to Preventing Deadlocks and Leaks in Go — Richard Fliam (video)
- Go GC: Solving the Latency Problem — Rick Hudson (video) (slides)
- Simplicity and Go — Katherine Cox-Buday (video) (slides)
- Rebuilding Parse.com in Go - an opinionated rewrite — Abhishek Kona (video) (slides)
- Prometheus: Designing and Implementing a Modern Monitoring Solution in Go — Björn Rabenstein (video) (slides)
- What Could Go Wrong? — Kevin Cantwell (video)
- The Roots of Go — Baishampayan Ghose (video) (slides)
- The Evolution of Go — Robert Griesemer (video) (slides)
- Static Code Analysis Using SSA — Ben Johnson (video) (slides)
- Go on Mobile — Hana Kim (video) (slides)
- Go Dynamic Tools — Dmitry Vyukov (video) (slides)
- Embrace the Interface — Tomás Senart (video) (slides)
- Uptime: Building Resilient Services with Go — Blake Caldwell (video) (slides)
- Cayley: Building a Graph Database — Barak Michener (video) (slides)
- Code Generation For The Sake Of Consistency — Sarah Adams (video)
- The Many Faces of Struct Tags — Sam Helman and Kyle Erf (video) (slides)
- Betting the Company on Go and Winning — Kelsey Hightower (video)
- How Go Was Made — Andrew Gerrand (video) (slides)
Huge thanks to the event organizers Brian Ketelsen and Eric St. Martin and their production team, the sponsors, the speakers, and the attendees for making this such a fun and action-packed conference. Hope to see you there next year!
See the index for more articles.