The battle for an IDE-friendly stack trace in Go (with and without Bazel)

Short description

Debugging in software development involves not only writing code but also making it error-free, and it is important for the process to be as comfortable as possible. This article discusses how to reconcile the call stack format and the IDE while using the Go programming language. The article covers the stack display options provided by the Go build, the convenience of different stack trace formats in the IDE, and various ways to get a call stack in the desired format. The article concludes by discussing the challenges with converting the stack before outputting and the changes observed after switching to assembly through Bazel.

The battle for an IDE-friendly stack trace in Go (with and without Bazel)

p align=”justify”> Software development involves not only writing code, but also debugging it. And debugging should be as comfortable as possible.

We write to the call stack beam with some errors. The IDE (Idea, GoLand) used by us allows you to get comfortable file navigation from the copied call stack (Analyze external stack traces). Unfortunately, this feature only works well if the binary is compiled on the same host where the IDE is running.

This post is about how we tried to reconcile the call stack format and the IDE.

And what are the stack display options provided by go build?

There are two handles in go build to affect the stack output format:

  • flag -trimpath – brings the display of the call stack to the same view, regardless of the local location of the files;

  • environment variable GOROOT_FINAL – allows you to replace the prefix to the system libraries in the stack when the flag is disabled -trimpath.

A stack display comparison program

Let’s consider the stack display on the example of a small program.

The source code can be downloaded at: https://github.com/bozaro/go-stack-trace

Actually, the program (stacktrace/main.go):

package main

import (
	"fmt"

	"github.com/Masterminds/cookoo"
	"github.com/pkg/errors"
)

func main() {
	// Build a new Cookoo app.
	registry, router, context := cookoo.Cookoo()
	// Fill the registry.
	registry.AddRoutes(
		cookoo.Route{
			Name: "TEST",
			Help: "A test route",
			Does: cookoo.Tasks{
				cookoo.Cmd{
					Name: "hi",
					Fn:   HelloWorld,
				},
			},
		},
	)
	// Execute the route.
	router.HandleRequest("TEST", context, false)
}

func HelloWorld(cxt cookoo.Context, params *cookoo.Params) (interface{}, cookoo.Interrupt) {
	fmt.Printf("%+v\n", errors.New("Hello World"))
	return true, nil
}

And small go.mod:

module github.com/bozaro/go-stack-trace

go 1.20

require (
	github.com/Masterminds/cookoo v1.3.0
	github.com/pkg/errors v0.9.1
)

Good old GOPATH

For the sake of order, let’s start with the old good GOPATH.

An example of a conclusion:

➜ GO111MODULE=off GOPATH=$(pwd) go get -d github.com/bozaro/go-stack-trace/stacktrace
➜ GO111MODULE=off GOPATH=$(pwd) go run github.com/bozaro/go-stack-trace/stacktrace 
Hello World
main.HelloWorld
	/home/bozaro/gopath/src/github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	/home/bozaro/gopath/src/github.com/Masterminds/cookoo/router.go:131
main.main
	/home/bozaro/gopath/src/github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	/usr/lib/go-1.20/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go-1.20/src/runtime/asm_amd64.s:1598

Everything is simple here: we see the full paths to each file.

At the same time, all paths are located either in src GoLang directory, or in a directory GOPATH.

Unfortunately, such a stack will only point to existing files if the file is compiled in an environment with the same directory layout. In our case, when part of the developers is MacOS, and the assembly for the combat environment is carried out under Linux, this requirement is impossible.

Fortunately, there is a flag -trimpathwhich cuts off the variable part from the call stack:

➜ GO111MODULE=off GOPATH=$(pwd) go run -trimpath github.com/bozaro/go-stack-trace/stacktrace
Hello World
main.HelloWorld
	github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	github.com/Masterminds/cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	github.com/Masterminds/cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	github.com/Masterminds/cookoo/router.go:131
main.main
	github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1598

In this case, all paths will be either relative GOPATHor regarding src in the GoLang directory.

It turned out a quite tolerable kind of challenge stack.

Go Modules

When using Go Modules, flag behavior -trimpath changes dramatically.

Let’s compare the output of the call stack without it:

➜ git clone https://github.com/bozaro/go-stack-trace.git .
➜ go run ./stacktrace 
Hello World
main.HelloWorld
	/home/bozaro/github/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	/home/bozaro/go/pkg/mod/github.com/!masterminds/[email protected]/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	/home/bozaro/go/pkg/mod/github.com/!masterminds/[email protected]/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	/home/bozaro/go/pkg/mod/github.com/!masterminds/[email protected]/router.go:131
main.main
	/home/bozaro/github/go-stack-trace/stacktrace/main.go:27
runtime.main
	/usr/lib/go-1.20/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go-1.20/src/runtime/asm_amd64.s:1598

And a similar conclusion from -trimpath:

➜ go run -trimpath ./stacktrace
Hello World
main.HelloWorld
	github.com/bozaro/go-stack-trace/stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	github.com/Masterminds/[email protected]/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	github.com/Masterminds/[email protected]/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	github.com/Masterminds/[email protected]/router.go:131
main.main
	github.com/bozaro/go-stack-trace/stacktrace/main.go:27
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1598

Without -trimpath we still see the full paths to each file. At the same time, we clearly trace three sources with source files:

  • working directory with the repository (in this example: $HOME/github/go-stack-trace);

  • GoLang system libraries from $GOROOT/src (In this example: /usr/lib/go-1.20/src);

  • external libraries from $GOMODCACHE (In this example: $HOME/go/pkg/mod);

At the same time, unlike GOPATH, the flag -trimpath does not cut off the prefix in file names, but otherwise forms it:

  1. files from the current module in the working directory are named with the name of the module from go.mod as a prefix (in this example: $HOME/github/go-stack-tracegithub.com/bozaro/go-stack-trace);

  2. GoLang system libraries from $GOROOT/src receive file names without a prefix;

  3. third-party libraries receive the name of the module with the version as a prefix (in this example: /home/bozaro/go/pkg/mod/github.com/!masterminds/[email protected]github.com/Masterminds/[email protected] – I pay special attention to the fact that the word Masterminds in the path to the file and the name of the module are written differently).

What stack trace is convenient for an IDE?

Unexpectedly, if you open the project from the repository in Idea/GoLand and try to parse any of the above call stacks, there will be no source file navigation:

  • call stack options for GOPATH are not suitable because this mini-project uses Go Modules and has a different file layout;

  • option for Go Modules without -trimpath won’t work because your home directory will most likely be different from /home/bozaro;

  • option for Go Modules with -trimpath will not work, because it is not supported in the IDE (https://youtrack.jetbrains.com/issue/GO-13827), and of all the paths visible in the stack, only files from the Go SDK will be found as suffixes of existing files.

From the side, it looks like the IDE in our case looks for source files by paths relative to the project directory and its parents.

As a result, a satisfactory format of the ported call stack is as follows:

main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	go/pkg/mod/github.com/!masterminds/[email protected]/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	go/pkg/mod/github.com/!masterminds/[email protected]/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	go/pkg/mod/github.com/!masterminds/[email protected]/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	GOROOT/src/runtime/asm_amd64.s:1598

That is:

  • project file paths are displayed relative to the project root;

  • as paths to third-party dependencies, the path to the module is used relative $GOMODCACHEbut with a prefix go/pkg/mod (The IDE will find this path when the project is in the home directory and environment variables GOPATH and GOMODCACHE have the value “default”);

  • as paths to files from the Go SDK we simply take the word GOROOT. We never managed to figure out a way for the IDE to find files like this without dancing around.

With this call stack format, the IDE recognizes all files except those from the Go SDK. The whole construct breaks if the developer locally overrides the environment variables GOPATH or GOMODCACHEBut I don’t know the scenarios when it is really needed.

How do I get the call stack in the right format?

I can see the following ways to get the call stack in the desired format:

  • affect the build so that the debug information contains the required file paths;

  • before output, convert the call stack into the desired format;

  • make an external utility that converts the call stack into the desired format.

Influence the assembly

We cannot influence the build in the case of Go Build to get the call stack format we want.

Stack conversion before outputting

In our case, we use the github.com/joomcode/errorx library, and it has a method to convert to the desired call stack format before outputting: https://pkg.go.dev/github.com/joomcode/errorx#InitializeStackTraceTransformer

Converting a path from a view without -trimpath looks trivial in its own right.

But this method has a number of disadvantages:

  • if the call stack passed this filter, it will remain in the original format;

  • some places, for example pprofare guaranteed to be transmitted in the original format.

External utility

Using an external utility greatly complicates the overall call stack analysis scenario.

In our case, in most cases, the stack was taken from leagues and there it was already in a convenient form, so we did not seriously consider this option.

The second round after switching to assembly through Bazel

In general, we were fine with converting the stack before leagues until switching to drafting via Bazel. But building through Bazel took the problem to a new level.

Call stack format after bazel assembly

➜ bazel run //stacktrace 
...
INFO: Running command line: bazel-bin/stacktrace/stacktrace_/stacktrace
Hello World
main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	external/com_github_masterminds_cookoo/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	external/com_github_masterminds_cookoo/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	external/com_github_masterminds_cookoo/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	src/runtime/asm_amd64.s:1598

We do not require developers to use build and run files through Bazel for a number of reasons. The main ones are:

  • we generate BUILD-files with its own utility and we do not want to require re-generation of files for every sneeze (it is fast, but not instantaneous);

  • IDE synchronization and BUILD-Files are quite slow.

At the same time, in the call stack from Bazel:

  • third-party libraries start referencing the external– a directory that the IDE does not see;

  • it is not possible to trivially obtain the path to the module in GOMODCACHE – information about the version of the module is lost;

  • generated files may receive a completely unexpected type prefix bazel-out/k8-fastbuild-ST-2df1151a1acb/....

All these paths refer to real files and are completely conscious in the context of Bazel, but without full integration they are only intimidating.

Stack conversion before outputting

At first, they tried to collect a set of rules that allow you to form something acceptable from the existing stack of calls.

For this through x_defsand then embed a separately generated file was passed to the program, which contained correspondence between the external name and the desired prefix in the call stack.

Also made a number of transformations to handle generated file paths.

The problem became less acute, but the result was still unsatisfactory:

  • in pprof complete horror remained;

  • some paths were converted incorrectly;

  • the whole construction as a whole was quite complex and fragile.

External utility

We didn’t want to go this way: in addition to all the complexity and fragility when transforming the stack before output, there was also the problem of subscribing to this utility the information that we sewed into the executable file, namely the correspondence of the external-name to the desired prefix in the call stack.

That is, in fact, it was supposed to be a call stack deobfusator, but this obfuscation itself only hindered us 🙁

Influence the build to require file paths

When using Bazel, the build is at a lower level than Go Build. There was hope to fix the assembly to have convenient file paths.

In utilities $(go env GOTOOLDIR)/compile is also a parameter -trimpath. But this parameter is no longer a boolean flag, but a list of prefix substitutions.

As a result, we added to the rules go_library and go_repository additional attributes to be able to influence the call stack:

After these changes, you can override the file path in the call stack, for example:

diff --git a/deps.bzl b/deps.bzl
index ffe4981..d917282 100644
--- a/deps.bzl
+++ b/deps.bzl
@@ -5,6 +5,7 @@ def go_dependencies():
         name = "com_github_masterminds_cookoo",
         importpath = "github.com/Masterminds/cookoo",
         sum = "h1:zwplWkfGEd4NxiL0iZHh5Jh1o25SUJTKWLfv2FkXh6o=",
+        stackpath = "go/pkg/mod/github.com/!masterminds/[email protected]",
         version = "v1.3.0",
     )
     go_repository(
@@ -12,4 +13,5 @@ def go_dependencies():
         importpath = "github.com/pkg/errors",
         sum = "h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=",
         version = "v0.9.1",
+        stackpath = "go/pkg/mod/github.com/pkg/[email protected]",
     )

An example of derivation in a branch bazel:

➜ git checkout bazel
➜ bazel run //stacktrace
INFO: Running command line: bazel-bin/stacktrace/stacktrace_/stacktrace
Hello World
main.HelloWorld
	stacktrace/main.go:31
github.com/Masterminds/cookoo.(*Router).doCommand
	go/pkg/mod/github.com/!masterminds/[email protected]/router.go:209
github.com/Masterminds/cookoo.(*Router).runRoute
	go/pkg/mod/github.com/!masterminds/[email protected]/router.go:164
github.com/Masterminds/cookoo.(*Router).HandleRequest
	go/pkg/mod/github.com/!masterminds/[email protected]/router.go:131
main.main
	stacktrace/main.go:27
runtime.main
	GOROOT/src/runtime/proc.go:250
runtime.goexit
	src/runtime/asm_amd64.s:1598

NOTE: The patch on Gazelle for some reason does not pick up by itself. If a type error occurs while running the example flag provided but not defined: -stack_path_prefix, then to fix it, you need to reassemble Gazelle itself. In this case, the easiest way to reset the Bazel cache is: bazel clean --expunge && bazel shutdown.

If you have comments, words of support (perhaps condemnation), or a desire to share experiences, we’d love to hear about your battles!

Related posts