The Go Blog

Smaller Go 1.7 binaries

18 August 2016

Introduction

Go was designed for writing servers. That is how it is most widely used today, and as a result a lot of work on the runtime and compiler is focused on issues that matter to servers: latency, ease of deployment, precise garbage collection, fast startup time, performance.

As Go gets used for a wider variety of programs, there are new issues that must be considered. One of these is binary size. It has been on the radar for a long time (issue #6853 was filed over two years ago), but the growing interest in using Go for deploying binaries on smaller devices — such as the Raspberry Pi or mobile devices — means it received some attention for the Go 1.7 release.

Work done in Go 1.7

Three significant changes in Go 1.7 affect binary size.

The first is the new SSA backend that was enabled for AMD64 in this release. While the primary motivation for SSA was improved performance, the better generated code is also smaller. The SSA backend shrinks Go binaries by ~5%. We expect larger gains for the more RISC-like architectures like ARM and MIPS when those backends have been converted to SSA in Go 1.8.

The second change is method pruning. Until 1.6, all methods on all used types were kept, even if some of the methods were never called. This is because they might be called through an interface, or called dynamically using the reflect package. Now the compiler discards any unexported methods that do not match an interface. Similarly the linker can discard other exported methods, those that are only accessible through reflection, if the corresponding reflection features are not used anywhere in the program. That change shrinks binaries by 5–20%.

The third change is a more compact format for run-time type information used by the reflect package. The encoding format was originally designed to make the decoder in the runtime and reflect packages as simple as possible. By making this code a bit harder to read we can compress the format without affecting the run-time performance of Go programs. The new format shrinks Go binaries by a further 5–15%. Libraries built for Android and archives built for iOS shrink further as the new format contains fewer pointers, each of which requires dynamic relocations in position independent code.

In addition, there were many small improvements such as improved interface data layout, better static data layout, and simplified dependencies. For example, the HTTP client no longer links in the entire HTTP server. The full list of changes can be found in issue #6853.

Results

Typical programs, ranging from tiny toys to large production programs, are about 30% smaller when built with Go 1.7.

The canonical hello world program goes from 2.3MB to 1.6MB:

package main

import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

When compiled without debugging information the statically linked binary is now under a megabyte.

A large production program used for testing this cycle, jujud, went from 94MB to 67MB.

Position-independent binaries are 50% smaller.

In a position-independent executable (PIE), a pointer in a read-only data section requires a dynamic relocation. Because the new format for type information replaces pointers by section offsets, it saves 28 bytes per pointer.

Position-independent executables with debugging information removed are particularly important to mobile developers, as this is the kind of program shipped to phones. Big downloads make for a poor user experience, so the reduction here is good news.

Future Work

Several changes to the run-time type information were too late for the Go 1.7 freeze, but will hopefully make it into 1.8, further shrinking programs, especially position-independent ones.

These changes are all conservative, reducing binary size without increasing build time, startup time, overall execution time, or memory usage. We could take more radical steps to reduce binary size: the upx tool for compressing executables shrinks binaries by another 50% at the cost of increased startup time and potentially increased memory use. For extremely small systems (the kind that might live on a keychain) we could build a version of Go without reflection, though it is unclear whether such a restricted language would be sufficiently useful. For some algorithms in the runtime we could use slower but more compact implementions when every kilobyte counts. All of these call for more research in later development cycles.

To the many contributors who helped make Go 1.7 binaries smaller, thank you!

By David Crawshaw

Go 1.7 is released

15 August 2016

Today we are happy to announce the release of Go 1.7. You can get it from the download page. There are several significant changes in this release: a port for Linux on IBM z Systems (s390x), compiler improvements, the addition of the context package, and support for hierarchical tests and benchmarks.

A new compiler back end, based on static single-assignment form (SSA), has been under development for the past year. By representing a program in SSA form, a compiler may perform advanced optimizations more easily. This new back end generates more compact, more efficient code that includes optimizations like bounds check elimination and common subexpression elimination. We observed a 5–35% speedup across our benchmarks. For now, the new backend is only available for the 64-bit x86 platform ("amd64"), but we’re planning to convert more architecture backends to SSA in future releases.

The compiler front end uses a new, more compact export data format, and processes import declarations more efficiently. While these changes across the compiler toolchain are mostly invisible, users have observed a significant speedup in compile time and a reduction in binary size by as much as 20–30%.

Programs should run a bit faster due to speedups in the garbage collector and optimizations in the standard library. Programs with many idle goroutines will experience much shorter garbage collection pauses than in Go 1.6.

Over the past few years, the golang.org/x/net/context package has proven to be essential to many Go applications. Contexts are used to great effect in applications related to networking, infrastructure, and microservices (such as Kubernetes and Docker). They make it easy to enable cancelation, timeouts, and passing request-scoped data. To make use of contexts within the standard library and to encourage more extensive use, the package has been moved from the x/net repository to the standard library as the context package. Support for contexts has been added to the net, net/http, and os/exec packages. For more information about contexts, see the package documentation and the Go blog post Go Concurrency Patterns: Context.

Go 1.5 introduced experimental support for a "vendor" directory, enabled by the GO15VENDOREXPERIMENT environment variable. Go 1.6 enabled this behavior by default, and in Go 1.7, this switch has been removed and the "vendor" behavior is always enabled.

Go 1.7 includes many more additions, improvements, and fixes. Find the complete set of changes, and details of the points above, in the Go 1.7 release notes.

Finally, the Go team would like thank everyone who contributed to the release. 170 people contributed to this release, including 140 from the Go community. These contributions ranged from changes to the compiler and linker, to the standard library, to documentation, and code reviews. We welcome contributions; if you'd like to get involved, check out the contribution guidelines.

By Chris Broadfoot

Go 1.6 is released

17 February 2016

Today we release Go version 1.6, the seventh major stable release of Go. You can grab it right now from the download page. Although the release of Go 1.5 six months ago contained dramatic implementation changes, this release is more incremental.

The most significant change is support for HTTP/2 in the net/http package. HTTP/2 is a new protocol, a follow-on to HTTP that has already seen widespread adoption by browser vendors and major websites. In Go 1.6, support for HTTP/2 is enabled by default for both servers and clients when using HTTPS, bringing the benefits of the new protocol to a wide range of Go projects, such as the popular Caddy web server.

The template packages have learned some new tricks, with support for trimming spaces around template actions to produce cleaner template output, and the introduction of the {{block}} action that can be used to create templates that build on other templates. A new template example program demonstrates these new features.

Go 1.5 introduced experimental support for a “vendor” directory that was enabled by an environment variable. In Go 1.6, the feature is now enabled by default. Source trees that contain a directory named “vendor” that is not used in accordance with the new feature will require changes to avoid broken builds (the simplest fix is to rename the directory).

The runtime has added lightweight, best-effort detection of concurrent misuse of maps. As always, if one goroutine is writing to a map, no other goroutine should be reading or writing the map concurrently. If the runtime detects this condition, it prints a diagnosis and crashes the program. The best way to find out more about the problem is to run it under the race detector, which will more reliably identify the race and give more detail.

The runtime has also changed how it prints program-ending panics. It now prints only the stack of the panicking goroutine, rather than all existing goroutines. This behavior can be configured using the GOTRACEBACK environment variable or by calling the debug.SetTraceback function.

Users of cgo should be aware of major changes to the rules for sharing pointers between Go and C code. The rules are designed to ensure that such C code can coexist with Go's garbage collector and are checked during program execution, so code may require changes to avoid crashes. See the release notes and cgo documentation for the details.

The compiler, linker, and go command have a new -msan flag analogous to -race and only available on linux/amd64, that enables interoperation with the Clang MemorySanitizer. This is useful for testing a program containing suspect C or C++ code. You might like to try it while testing your cgo code with the new pointer rules.

Performance of Go programs built with Go 1.6 remains similar to those built with Go 1.5. Garbage-collection pauses are even lower than with Go 1.5, but this is particularly noticeable for programs using large amounts of memory. With regard to the performance of the compiler tool chain, build times should be similar to those of Go 1.5.

The algorithm inside sort.Sort was improved to run about 10% faster, but the change may break programs that expect a specific ordering of equal but distinguishable elements. Such programs should refine their Less methods to indicate the desired ordering or use sort.Stable to preserve the input order for equal values.

And, of course, there are many more additions, improvements, and fixes. You can find them all in the comprehensive release notes.

To celebrate the release, Go User Groups around the world are holding release parties on the 17th of February. Online, the Go contributors are hosting a question and answer session on the golang subreddit for the next 24 hours. If you have questions about the project, the release, or just Go in general, then please join the discussion.

Thanks to everyone that contributed to the release. Happy hacking.

By Andrew Gerrand

Language and Locale Matching in Go

9 February 2016

Introduction

Consider an application, such as a web site, with support for multiple languages in its user interface. When a user arrives with a list of preferred languages, the application must decide which language it should use in its presentation to the user. This requires finding the best match between the languages the application supports and those the user prefers. This post explains why this is a difficult decision and how Go can help.

Language Tags

Language tags, also known as locale identifiers, are machine-readable identifiers for the language and/or dialect being used. The most common reference for them is the IETF BCP 47 standard, and that is the standard the Go libraries follow. Here are some examples of BCP 47 language tags and the language or dialect they represent.

Tag Description
en English
en-US American English
cmn Mandarin Chinese
zh Chinese, typically Mandarin
nl Dutch
nl-BE Flemish
es-419 Latin American Spanish
az, az-Latn both Azerbaijani written in Latin script
az-Arab Azerbaijani written in Arabic

The general form of the language tag is a language code (“en”, “cmn”, “zh”, “nl”, “az” above) followed by an optional subtag for script (“-Arab”), region (“-US”, “-BE”, “-419”), variants (“-oxendict” for Oxford English Dictionary spelling), and extensions (“-u-co-phonebk” for phone-book sorting). The most common form is assumed if a subtag is omitted, for instance “az-Latn-AZ” for “az”.

The most common use of language tags is to select from a set of system-supported languages according to a list of the user's language preferences, for example deciding that a user who prefers Afrikaans would be best served (assuming Afrikaans is not available) by the system showing Dutch. Resolving such matches involves consulting data on mutual language comprehensibility.

The tag resulting from this match is subsequently used to obtain language-specific resources such as translations, sorting order, and casing algorithms. This involves a different kind of matching. For example, as there is no specific sorting order for Portuguese, a collate package may fall back to the sorting order for the default, or “root”, language.

The Messy Nature of Matching Languages

Handling language tags is tricky. This is partly because the boundaries of human languages are not well defined and partly because of the legacy of evolving language tag standards. In this section we will show some of the messy aspects of handling language tags.

Tags with different language codes can indicate the same language

For historical and political reasons, many language codes have changed over time, leaving languages with an older legacy code as well as a new one. But even two current codes may refer to the same language. For example, the official language code for Mandarin is “cmn”, but “zh” is by far the most commonly used designator for this language. The code “zh” is officially reserved for a so called macro language, identifying the group of Chinese languages. Tags for macro languages are often used interchangeably with the most-spoken language in the group.

Matching language code alone is not sufficient

Azerbaijani (“az”), for example, is written in different scripts depending on the country in which it is spoken: "az-Latn" for Latin (the default script), "az-Arab" for Arabic, and “az-Cyrl” for Cyrillic. If you replace "az-Arab" with just "az", the result will be in Latin script and may not be understandable to a user who only knows the Arabic form.

Also different regions may imply different scripts. For example: “zh-TW” and “zh-SG” respectively imply the use of Traditional and Simplified Han. As another example, “sr” (Serbian) defaults to Cyrillic script, but “sr-RU” (Serbian as written in Russia) implies the Latin script! A similar thing can be said for Kyrgyz and other languages.

If you ignore subtags, you might as well present Greek to the user.

The best match might be a language not listed by the user

The most common written form of Norwegian (“nb”) looks an awful lot like Danish. If Norwegian is not available, Danish may be a good second choice. Similarly, a user requesting Swiss German (“gsw”) will likely be happy to be presented German (“de”), though the converse is far from true. A user requesting Uygur may be happier to fall back to Chinese than to English. Other examples abound. If a user-requested language is not supported, falling back to English is often not the best thing to do.

The choice of language decides more than translation

Suppose a user asks for Danish, with German as a second choice. If an application chooses German, it must not only use German translations but also use German (not Danish) collation. Otherwise, for example, a list of animals might sort “Bär” before “Äffin”.

Selecting a supported language given the user’s preferred languages is like a handshaking algorithm: first you determine which protocol to communicate in (the language) and then you stick with this protocol for all communication for the duration of a session.

Using a “parent” of a language as fallback is non-trivial

Suppose your application supports Angolan Portuguese (“pt-AO”). Packages in golang.org/x/text, like collation and display, may not have specific support for this dialect. The correct course of action in such cases is to match the closest parent dialect. Languages are arranged in a hierarchy, with each specific language having a more general parent. For example, the parent of “en-GB-oxendict” is “en-GB”, whose parent is “en”, whose parent is the undefined language “und”, also known as the root language. In the case of collation, there is no specific collation order for Portugese, so the collate package will select the sorting order of the root language. The closest parent to Angolan Portuguese supported by the display package is European Portuguese (“pt-PT”) and not the more obvious “pt”, which implies Brazilian.

In general, parent relationships are non-trivial. To give a few more examples, the parent of “es-CL” is “es-419”, the parent of “zh-TW” is “zh-Hant”, and the parent of “zh-Hant” is “und”. If you compute the parent by simply removing subtags, you may select a “dialect” that is incomprehensible to the user.

Language Matching in Go

The Go package golang.org/x/text/language implements the BCP 47 standard for language tags and adds support for deciding which language to use based on data published in the Unicode Common Locale Data Repository (CLDR).

Here is a sample program, explained below, matching a user's language preferences against an application's supported languages:

package main

import (
    "fmt"

    "golang.org/x/text/language"
    "golang.org/x/text/language/display"
)

var userPrefs = []language.Tag{
    language.Make("gsw"), // Swiss German
    language.Make("fr"),  // French
}

var serverLangs = []language.Tag{
    language.AmericanEnglish, // en-US fallback
    language.German,          // de
}

var matcher = language.NewMatcher(serverLangs)

func main() {
    tag, index, confidence := matcher.Match(userPrefs...)

    fmt.Printf("best match: %s (%s) index=%d confidence=%v\n",
        display.English.Tags().Name(tag),
        display.Self.Name(tag),
        index, confidence)
    // best match: German (Deutsch) index=1 confidence=High
}

Creating Language Tags

The simplest way to create a language.Tag from a user-given language code string is with language.Make. It extracts meaningful information even from malformed input. For example, “en-USD” will result in “en” even though USD is not a valid subtag.

Make doesn’t return an error. It is common practice to use the default language if an error occurs anyway so this makes it more convenient. Use Parse to handle any error manually.

The HTTP Accept-Language header is often used to pass a user’s desired languages. The ParseAcceptLanguage function parses it into a slice of language tags, ordered by preference.

By default, the language package does not canonicalize tags. For example, it does not follow the BCP 47 recommendation of eliminating scripts if it is the common choice in the “overwhelming majority”. It similarly ignores CLDR recommendations: “cmn” is not replaced by “zh” and “zh-Hant-HK” is not simplified to “zh-HK”. Canonicalizing tags may throw away useful information about user intent. Canonicalization is handled in the Matcher instead. A full array of canonicalization options are available if the programmer still desires to do so.

Matching User-Preferred Languages to Supported Languages

A Matcher matches user-preferred languages to supported languages. Users are strongly advised to use it if they don’t want to deal with all the intricacies of matching languages.

The Match method may pass through user settings (from BCP 47 extensions) from the preferred tags to the selected supported tag. It is therefore important that the tag returned by Match is used to obtain language-specific resources. For example, “de-u-co-phonebk” requests phone-book ordering for German. The extension is ignored for matching, but is used by the collate package to select the respective sorting order variant.

A Matcher is initialized with the languages supported by an application, which are usually the languages for which there are translations. This set is typically fixed, allowing a matcher to be created at startup. Matcher is optimized to improve the performance of Match at the expense of initialization cost.

The language package provides a predefined set of the most commonly used language tags that can be used for defining the supported set. Users generally don’t have to worry about the exact tags to pick for supported languages. For example, AmericanEnglish (“en-US”) may be used interchangeably with the more common English (“en”), which defaults to American. It is all the same for the Matcher. An application may even add both, allowing for more specific American slang for “en-US”.

Matching Example

Consider the following Matcher and lists of supported languages:

var supported = []language.Tag{
    language.AmericanEnglish,    // en-US: first language is fallback
    language.German,             // de
    language.Dutch,              // nl
    language.Portuguese          // pt (defaults to Brazilian)
    language.EuropeanPortuguese, // pt-pT
    language.Romanian            // ro
    language.Serbian,            // sr (defaults to Cyrillic script)
    language.SerbianLatin,       // sr-Latn
    language.SimplifiedChinese,  // zh-Hans
    language.TraditionalChinese, // zh-Hant
}
var matcher = language.NewMatcher(supported)

Let's look at the matches against this list of supported languages for various user preferences.

For a user preference of "he" (Hebrew), the best match is "en-US" (American English). There is no good match, so the matcher uses the fallback language (the first in the supported list).

For a user preference of "hr" (Croatian), the best match is "sr-Latn" (Serbian with Latin script), because, once they are written in the same script, Serbian and Croatian are mutually intelligible.

For a user preference of "ru, mo" (Russian, then Moldavian), the best match is "ro" (Romanian), because Moldavian is now canonically classified as "ro-MD" (Romanian in Moldova).

For a user preference of "zh-TW" (Mandarin in Taiwan), the best match is "zh-Hant" (Mandarin written in Traditional Chinese), not "zh-Hans" (Mandarin written in Simplified Chinese).

For a user preference of "af, ar" (Afrikaans, then Arabic), the best match is "nl" (Dutch). Neither preference is supported directly, but Dutch is a significantly closer match to Afrikaans than the fallback language English is to either.

For a user preference of "pt-AO, id" (Angolan Portuguese, then Indonesian), the best match is "pt-PT" (European Portuguese), not "pt" (Brazilian Portuguese).

For a user preference of "gsw-u-co-phonebk" (Swiss German with phone-book collation order), the best match is "de-u-co-phonebk" (German with phone-book collation order). German is the best match for Swiss German in the server's language list, and the option for phone-book collation order has been carried over.

Confidence Scores

Go uses coarse-grained confidence scoring with rule-based elimination. A match is classified as Exact, High (not exact, but no known ambiguity), Low (probably the correct match, but maybe not), or No. In case of multiple matches, there is a set of tie-breaking rules that are executed in order. The first match is returned in the case of multiple equal matches. These confidence scores may be useful, for example, to reject relatively weak matches. They are also used to score, for example, the most likely region or script from a language tag.

Implementations in other languages often use more fine-grained, variable-scale scoring. We found that using coarse-grained scoring in the Go implementation ended up simpler to implement, more maintainable, and faster, meaning that we could handle more rules.

Displaying Supported Languages

The golang.org/x/text/language/display package allows naming language tags in many languages. It also contains a “Self” namer for displaying a tag in its own language.

For example:

    var supported = []language.Tag{
        language.English,            // en
        language.French,             // fr
        language.Dutch,              // nl
        language.Make("nl-BE"),      // nl-BE
        language.SimplifiedChinese,  // zh-Hans
        language.TraditionalChinese, // zh-Hant
        language.Russian,            // ru
    }

    en := display.English.Tags()
    for _, t := range supported {
        fmt.Printf("%-20s (%s)\n", en.Name(t), display.Self.Name(t))
    }

prints

English              (English)
French               (français)
Dutch                (Nederlands)
Flemish              (Vlaams)
Simplified Chinese   (简体中文)
Traditional Chinese  (繁體中文)
Russian              (русский)

In the second column, note the differences in capitalization, reflecting the rules of the respective language.

Conclusion

At first glance, language tags look like nicely structured data, but because they describe human languages, the structure of relationships between language tags is actually quite complex. It is often tempting, especially for English-speaking programmers, to write ad-hoc language matching using nothing other than string manipulation of the language tags. As described above, this can produce awful results.

Go's golang.org/x/text/language package solves this complex problem while still presenting a simple, easy-to-use API. Enjoy.

By Marcel van Lohuizen

Six years of Go

10 November 2015

Six years ago today the Go language was released as an open source project. Since then, more than 780 contributors have made over 30,000 commits to the project's 22 repositories. The ecosystem continues to grow, with GitHub reporting more than 90,000 Go repositories. And, offline, we see new Go events and user groups pop up around the world with regularity.

In August we released Go 1.5, the most significant release since Go 1. It features a completely redesigned garbage collector that makes the language more suitable for latency-sensitive applications; it marks the transition from a C-based compiler tool chain to one written entirely in Go; and it includes ports to new architectures, with better support for ARM processors (the chips that power most smartphones). These improvements make Go better suited to a broader range of tasks, a trend that we hope will continue over the coming years.

Improvements to tools continue to boost developer productivity. We introduced the execution tracer and the "go doc" command, as well as more enhancements to our various static analysis tools. We are also working on an official Go plugin for Sublime Text, with better support for other editors in the pipeline.

Early next year we will release more improvements in Go 1.6, including HTTP/2 support for net/http servers and clients, an official package vendoring mechanism, support for blocks in text and HTML templates, a memory sanitizer that checks both Go and C/C++ code, and the usual assortment of other improvements and fixes.

This is the sixth time we have had the pleasure of writing a birthday blog post for Go, and we would not be doing so if not for the wonderful and passionate people in our community. The Go team would like to thank everyone who has contributed code, written an open source library, authored a blog post, helped a new gopher, or just given Go a try. Without you, Go would not be as complete, useful, or successful as it is today. Thank you, and celebrate!

By Andrew Gerrand

See the index for more articles.