21 March 2019
Go 1.11 and Go 1.12 make significant progress toward allowing developers
to debug the same optimized binaries that they deploy to production.
As the Go compiler has become increasingly aggressive in producing faster binaries,
we've lost ground in debuggability.
In Go 1.10, users needed to disable optimizations entirely in order to have
a good debugging experience from interactive tools like Delve.
But users shouldn’t have to trade performance for debuggability,
especially when running production services.
If your problem is occurring in production,
you need to debug it in production, and that shouldn’t require deploying
For Go 1.11 and 1.12, we focused on improving the debugging experience on
optimized binaries (the default setting of the Go compiler).
Delve is a debugger for Go on x86
supporting both Linux and macOS.
Delve is aware of goroutines and other Go features and provides one of the
best Go debugging experiences.
Delve is also the debugging engine behind GoLand,
Delve normally rebuilds the code it is debugging with -gcflags "all=-N -l",
which disables inlining and most optimizations.
To debug optimized code with delve, first build the optimized binary,
then use dlv exec your_program to debug it.
Or, if you have a core file from a crash,
you can examine it with dlv core your_program your_core.
With 1.12 and the latest Delve releases, you should be able to examine many variables,
even in optimized binaries.
-gcflags "all=-N -l"
dlv exec your_program
dlv core your_program your_core
When debugging optimized binaries produced by Go 1.10,
variable values were usually completely unavailable.
In contrast, starting with Go 1.11, variables can usually be examined even
in optimized binaries,
unless they’ve been optimized away completely.
In Go 1.11 the compiler began emitting DWARF location lists so debuggers
can track variables as they move in and out of registers and reconstruct
complex objects that are split across different registers and stack slots.
This shows an example of stepping through a simple function in a debugger in 1.10,
with flaws (skipped and repeated lines) highlighted by red arrows.
Flaws like this make it easy to lose track of where you are when stepping
through a program and interfere with hitting breakpoints.
Go 1.11 and 1.12 record statement boundary information and do a better job
of tracking source line numbers through optimizations and inlining.
As a result, in Go 1.12, stepping through this code stops on every line
and does so in the order you would expect.
Function call support in Delve is still under development, but simple cases work. For example:
(dlv) call fib(6)
> main.main() ./hello.go:15 (PC: 0x49d648)
Go 1.12 is a step toward a better debugging experience for optimized binaries
and we have plans to improve it even further.
There are fundamental tradeoffs between debuggability and performance,
so we’re focusing on the highest-priority debugging defects,
and working to collect automated metrics to monitor our progress and catch regressions.
We’re focusing on generating correct information for debuggers about variable locations,
so if a variable can be printed, it is printed correctly.
We’re also looking at making variable values available more of the time,
particularly at key points like call sites,
though in many cases improving this would require slowing down program execution.
Finally, we’re working on improving stepping:
we’re focusing on the order of stepping with panics,
the order of stepping around loops, and generally trying to follow source
order where possible.
Go 1.11 started compressing debug information to reduce binary sizes.
This is natively supported by Delve, but neither LLDB nor GDB support compressed
debug info on macOS.
If you are using LLDB or GDB, there are two workarounds:
build binaries with -ldflags=-compressdwarf=false,
or use splitdwarf
(go get golang.org/x/tools/cmd/splitdwarf) to decompress the debug information
in an existing binary.
go get golang.org/x/tools/cmd/splitdwarf