Demystifying Go proxy and Checksum Database

I have been working with Golang for the past few years now. The topic that always rattled me was the go.sum file, checksum database and their underlying concepts.

So, what I always knew was that they always work behind the scenes. These concepts guarantee that the Go dependencies that I download in my project today, will be the same, as what I will download a month from now. It can be any other environment or server. But, few questions always remained unanswered

How does Go ensure this secure behavior?
What would happen if the dependency I am using today disappeared tomorrow from the origin server, like GitHub?
What happens if an attacker intercepts and alters the dependency that I try to download from the internet?

This article aims to answer the above questions that you may have grappled with at some point working with Golang.

Go modules

Let’s begin with Go modules. Simply put, a Go module is a collection of related packages that work together. A module is defined in a go.mod file, which lists the dependencies your codebase requires. It looks like this:

module go-http-server

go 1.24.2

require (
    github.com/google/uuid v1.6.0
    github.com/gorilla/mux v1.8.1
    github.com/stretchr/testify v1.10.0
)

require (
    github.com/davecgh/go-spew v1.1.1 // indirect
    github.com/lib/pq v1.10.9 // indirect
    github.com/pmezard/go-difflib v1.0.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
)

Now, as you can see, for each of the dependency we specify a semantic version which goes by the below format

As indicated in the image, major version is incremented when there is any breaking change. E.g., in cases where the interfaces are changed, then the consumers of the service should update their dependency version to receive the latest changes. Minor version is incremented when there is any new feature or features added and is generally non-breaking in nature which means its backward compatible. Patch version is again non-breaking in nature and is incremented when we fix smaller bugs, or make any minor improvements in the code, that does not affect its previously agreed functionality.

So, lets now go back to the dependencies defined in go.mod. These modules must be available locally for the go module to execute successfully. In case, your project has multiple modules and they require different semantic versions of the same dependency, Go selects the minimum version that will satisfy the needs of all the modules. This algorithm is called as Minimum Version Selection(MVS). MVS traverses the graph and selects the highest required version or the minimum required version to get all packages to work.

Module Proxies

If you are a Golang developer, you have definitely come across go get command.

The primary function of command go get is to help us with fetching the source code of the dependency and resolve all its underlying dependencies. So now, imagine that go get needs to download the dependencies from a location where the code is hosted – like Github, Gitlab or BitBucket server.

Based on what we assume, the flow may look something like below :

Assuming this is the way go command fetches the dependencies, imagine what happens if the author of the dependency deleted the entire repository from Github OR the author updates/deletes the version that you are currently using in your Go module. And to make matters even more worse, what about the high latency that you will likely experience while building the app, since you are downloading all of your dependencies located at different geographic locations ?

Golang has an elegant solution to eliminate these problems for us – using proxies or mirrors.

A proxy server in general, is an intermediate server between a client and multiple servers. The proxy server forwards the request from the client to multiple servers. It can also cache any data in between.

Starting with Go1.13, a default GOPROXY is provided to us upon Golang installation.

This public GOPROXY is a centralized repository available to all the Golang developers for free. It hosts all the open-source Go modules made available from other authors and which are present in publicly accessible version control repositories.

The interaction of Go with the proxy looks like below:

Lets break down this interaction a little more for better understanding :

Lets say we need the module github.com/google/uuid to be added as the dependency in our Go project. When we do a go get, below actions occur behind the scenes :

go get tries to get the list of version for the requested module first from the proxy
Once go command receives all the versions, the latest version v1.6.0 is selected to be downloaded list. This is because we have not mentioned a specific version to be downloaded.
The command again asks for the metadata of v1.6.0 which contain info like the latest version, commit reference, time stamps etc
After receiving the metadata, the go command queries for the go.mod file of the module v1.6.0 to resolve all the transitive dependencies of the module
Once all the dependencies are resolved, the go command asks for the actual source code zip and downloads it locally.

The API spec of the proxy itself is relatively simple to use. You can run the commands easily on the terminal to see what the available versions of the module in the go proxy, or you can retrieve the metadata of the version and so on. Few examples are given below:

$ curl https://proxy.golang.org/github.com/google/uuid/@v/list

v1.3.0
v1.6.0
v1.0.0
v1.1.1
v1.3.1
v1.5.0
v1.1.3
v1.1.2
v1.1.5
v1.4.0
v1.1.4
v1.2.0
v1.1.0

$ curl https://proxy.golang.org/github.com/google/uuid/@v/v1.6.0.info

{"Version":"v1.6.0","Time":"2024-01-23T18:54:04Z","Origin":{"VCS":"git","URL":"https://github.com/google/uuid","Ref":"refs/tags/v1.6.0","Hash":"0f11ee6918f41a04c201eceeadf612a377bc7fbc"}}

In Golang we call the proxy as mirrors. A mirror is a special type of proxy that caches the metadata, source code and has its own implicit storage system. By default, the go command uses the Go module mirror https://proxy.golang.org

To use this module mirror, set the Golang environment variable to its URL:

$ export GOPROXY=https://proxy.golang.org

The above setting redirects all module download requests to the proxy server https://proxy.golang.org which is maintained by the Golang team.

Using GOPROXY, go get interacts with the proxy server instead of the origin server which we do not have any control on. Hence, by using GOPROXY we are achieving the fundamental requirements of immutability and availability of the dependency.

But what about private Go modules ?

A golang project may contain both public and private Go modules as dependencies. The private modules may be residing in the internal company servers and requires to be downloaded from those servers only. We do not want to reach out to proxy.golang.org for this purpose. In this case, we could use the environment variables GOPRIVATE or GONOPROXY that we can use to control the packages that is to be downloaded from internal servers, bypassing the GOPROXY.

Checksum Database

The next question is how do we ensure that we are downloading the correct version of the dependency ?

The answer lies in go.sum and checksum database.

You may be familiar with go.sum file if you are working on Go applications. They reside next to go.mod file typically in the root directory of the module. This file is automatically generated when you run a command like go mod tidy, and its not supposed to be updated by us, unless we have any merge conflicts.

This file contains cryptographic hashes of all the direct and transitive dependencies. It looks something like below :

go.sum file acts as a source of truth for anyone who uses the Go project and guarantees that the same versions of dependencies are downloaded every time. If the go.sum file changes without any dependency version changes, then it means the source code is not what we need. It could also mean the proxy server or origin server is hacked. This could indicate that you may have downloaded a malicious code into your project.

So how does Go authenticate whether a particular download is correct? In this context, “correct” means the same code is getting downloaded every time and for everyone in every machine.

For this, Go team at Google maintains a global source of go.sum lines called a checksum database. The checksum database runs at https://sum.golang.org/ and is built on a Transparent Log. A transparent log is a merkle tree structure that stores the cryptographic hashes of all the Go module versions.

A merkle tree structure is immutable/append-only and hence a trustworthy solution, as it cannot be tampered with. Using this data structure ensures the integrity and authenticity of Go modules. There are several use cases of Transparent logs especially in permissionless systems like blockchain, monitoring, compliance etc.

When the proxy server returns the zip file, go command computes an SHA256 cryptographic hash of the downloaded module. These hashes are validated against the existing go.sum file in your main module. If the hashes don’t match, go command reports an error. It also deletes the downloaded file without adding into the module cache. If the hashes are matched, then these are added in your projects go.sum file and the module is added into the module cache.

If go.sum does not exist, the go command verifies the hash of the downloaded zip file with the global checksum database or the transparent log. The go command verifies that the specific module version hash is actually present in the log, and ensures the integrity of the entire log, in order to detect any tampering.

Once the hash is verified against this database, its added into the go.sum file in the main module.

All these happens on the fly when you run a go get command on the terminal. The Go checksum database also serves endpoints that we could use to verify the go.sum lines against the database.

Conclusion

In the ever-evolving world of software development, consistency and security of dependencies are vital—and Go’s approach delivers both with elegance. By harnessing the power of go.sum, the checksum database, and the proxy architecture, the Go ecosystem ensures that developers can rely on immutable, verifiable dependencies across environments and over time.