Ysco - managed automated updates for Go services

There’s a new tool in town! It’s called ysco, and it provides managed automated updates for Go services, without requiring changes to the application.

The launch party was at the Go Rotterdam meetup earlier this evening. Special thanks to the organizers for running a well-organized & fun event, and bringing the Go community together. This post follows the same structure as the talk, with some additional information.

Ysco in a nutshell

Instead of starting your services like this (from a systemd unit file, or rc.d script):

./app [appflags]

You’ll start it like this:

./ysco run [yscoflags] ./app [appflags]

Ysco will start the application and periodically check for updates in the background. Once an update is found, depending on the configuration/policy, ysco can fetch a new binary, stop the application (by sending a TERM signal and waiting), replace the binary, and restart the application.

You are in control over which updates get installed, and when. That’s why ysco does “managed automated updates”, not just “automatic updates”. If you don’t want to install new updates automatically, you can set up notifications for available updates, and update at your convenience with a single click through the ysco admin web interface.

The magic is in how ysco knows when updates are available, and how it gets a binary for a new version. Before getting into those details, let’s take a step back and look at what “a new version” means.

New versions

We don’t just care about new releases of “app”. Just as important are new Go toolchain releases: They often come with stability & security fixes for the compiler, runtime and standard library. The statically linked application contains it all. Projects often publish binaries when they release a new version of their application. But they typically do not release new binaries (or docker images) when new Go toolchains are released. For internal services seeing active development and continuous delivery, updates will probably get deployed soon enough. But services that are feature-complete and you’re simply using (those are the best!) may not get updated as often as they should, too much hassle. Ysco is here to help! But there are some limitations to consider…

Requirements

Ysco only works for services, not one-off cli tools (it looks for updates in the background). Otherwise, many Go applications will fulfill the requirements:

• Buildable with CGO_ENABLED=0. It must not need cgo, so it must not link against any C libraries (unsafe! bad!1), which has been recommended practice for a long time.
• Self-contained, not needing external files. Common practice nowadays with the “embed” package and go:embed directives that include arbitrary files in binaries.
• Source code should be available through the Go module proxy. This is not a strict requirement. You can use ysco with private source code repositories, but you’ll have to run some additional infrastructure. It’s out of scope for this post. If you want to know how, look at the ysco integration tests.

How does ysco find updates for a Go application? Read on.

Finding updates

Go applications are just Go modules, and versioned and distributed just like library Go modules/packages. You can find the Go module path and version, and the Go toolchain version used, by running go version -m <binary>, or with the debug/buildinfo#ReadFile method. Go modules are available through the Go module proxy at https://proxy.golang.org, and each module is included in the Go sum database at https://sum.golang.org. For a Go module (application) at github.com/mjl-/moxtools, the Go module proxy will return all versions at: https://proxy.golang.org/github.com/mjl-/moxtools/@v/list. Get just the latest version at https://proxy.golang.org/github.com/mjl-/moxtools/@latest.

Go toolchain versions can also be retrieved through the Go module proxy. The module to look for is golang.org/toolchain. See https://proxy.golang.org/golang.org/toolchain/@v/list (about 100KB at the time of writing). It contains all releases, for all platforms. For example v0.0.1-go1.11.9.freebsd-amd64. The initial “v0.0.1” is fixed and to be ignored. The actual version comes next, go1.11.9. Each toolchain can be downloaded as a zip file (at the .../@v/<version>.zip Go module proxy endpoint), extracted, and run. Each module in the Go module proxy is included in the Go sum database, meaning you can verify their contents through its transparency log. See “Transparent Logs for Skeptical Clients” by Russ Cox at https://research.swtch.com/tlog for details and security properties of transparency logs.

Alternatively, you can find new Go toolchains through the Go website. The human readable HTML page is at https://go.dev/dl/, with a machine-readable version at https://go.dev/dl/?mode=json, with only the stable versions. If you add `&include=all, you’ll get all historic toolchains too (about 1MB of JSON at the time of writing). Download the referenced .tar.gz and .zip files. Verify the PGP signature at the same URL with .asc appended.

The Go module proxy will work fine for finding updates. But it will fetch some potentially largish HTTP responses. Can we do it with less overhead? Yes, we can! Earlier this year I created a service at https://www.gopherwatch.org that monitors additions to the Go sum database, sends out notifications for matching subscriptions (by email or webhook) and stores all modules and versions in its database. I also happened to have a growing interest in DNS. So I decided to add a DNS server to gopherwatch. You can now do DNS requests (lightweight!) to gopherwatch to discover the latest version of any Go module, with special support for the Go toolchains. It even implements DNSSEC with online signing with compact denial of existence. Example:

$ host -t txt toolchain.v0.l.gopherwatch.org
toolchain.v0.l.gopherwatch.org descriptive text "v=go1.23.1 k=cur t=66d9cf47; v=go1.22.7 k=prev t=66d9cf47"

$ host -t txt mox.mjl_2d._.github.com.v0.l.gopherwatch.org
mox.mjl_2d._.github.com.v0.l.gopherwatch.org descriptive text "v=v0.0.11 t=6631463d"

Zone l.gopherwatch.org is delegated to the gopherwatch instance. The current “DNS API” is v0. New versions may be defined in the future. Then there’s either the special toolchain, or an encoded Go module path. A Go module path starts with its original hostname. The _ before it is the host/path-separator. The labels before it are path elements with each special character escaped with a _ followed by its hexadecimal-encoded byte. The response has semicolon-separated “key=value” pairs, with v the latest version, t a hexadecimal unix epoch timestamp. For Go toolchains, k is the kind (cur, prev, next).

Ysco will check for updates at gopherwatch DNS by default, with fallback to the Go module proxy.

Policies & schedules

By default, patch-level updates are scheduled for automatic installation (with a small delay and jitter). In case of a new Go module application, the latest Go toolchain is used. By default, ysco keeps applications on supported Go toolchains, so after a maximum of a year without an application release, ysco will update to a more recent minor release. Applications would likely have seen a new release in a year if new Go toolchains required changes. In my experience, Go toolchain patch-level releases are very unlikely to break existing applications.

If you’re not interested in automatic updates, you can set a “manual” policy. Discovery of new versions is registered in prometheus metrics regardless. You can use monitoring/alerting tools to keep track of available updates. The optional ysco admin web interface has a button that installs an update. Just a single click.

If you want updates to be installed automatically, you can still influence when that happens, with a schedule. Perhaps only during working hours, or only evenings or weekends.

Fetching new binaries

Ysco will “fetch” a new binary when a new version of a Go application or Go toolchain is released. How does it do that?

The ysco requirements (non-cgo self-contained go-installable through Go module proxy) make it easy to build a binary. It boils down to running "CGO_ENABLED=0 go install <module>@<version>". But that requires a specific version of go to be installed. As we’ve seen, it’s not hard to fetch a Go toolchain. But managing toolchains and build/module caches is a bit of hassle on a server. Can’t we get a service to build binaries for us? Well yes, and I happen to be running one: https://beta.gobuilds.org. It automatically fetches Go toolchains as needed. It will cross-compile any version of any Go module to any platform (os/arch) supported by the Go toolchains on demand, or serve it from its cache.

Downloading binaries from a random service and running them locally may seem a bit unsafe. Why would you trust those binaries? Valid question.

First, the gobuilds service builds the binaries in a reproducible way. You can (manually) do a build locally and verify the result is the exact same binary, byte-for-byte identical. That should always be the case because the gobuilds service builds each binary twice, on different machines, with different architecture/configuration (this once found a minor reproducibility bug in the early days of Go reproducible builds).

Second, the hashes of the built binaries are included in the gobuilds transparency log (like the Go sum database), which ysco verifies.

Third, and you’re encouraged to do this, you can run your own instance of gobuilds, configure it to build its own binaries, and verify the result against the public gobuilds.org instance. You get to verify your binaries are reproducible, and you’ll help check the honesty of the public gobuilds.org service. Your instance would have its own transparency log verifier key that you would configure in ysco.

Road to here

How did it comes to this? Gobuilds.org has existed for many years. It was a fun excuse to play with transparency logs. I wrote gopherwatch earlier this year as a testbed for mox, my mail server. The process helped with designing & developing its webapi for sending email and receiving delivery feedback. It also gets to send notification emails to many people, exercising its outgoing delivery code. With gopherwatch already operating, adding a DNS server wasn’t a big leap.

I wanted mox to be easy to use and run from the start, it’s one of the project goals. Make it too much effort to run your own infrastructure and people will stop doing it. I included a simple DNS-based mechanism to check for updates once a day, on opt-in basis. Aside: One nice aspect of using DNS is that requests are often going through a network-wide recursive resolver, so individual machines can’t necessarily be identified, there’s caching, redundancy. When I release a new version of mox, I update the DNS TXT record at _updates.xmox.nl. I’ve done that 13 times now, ripe for automation. It now works for mox, and all other Go modules.

Another issue with running your own infrastructure is the effort it takes to update. Mox admins are already referred to gobuilds.org to download up-to-date binaries. But even for me it feels like a bit too much hassle to update promptly when needed. I had been pondering adding a one-click update mechanism. Ysco should do the trick, for mox and many more applications.

Discussion & caveats

Ysco is young and there’s plenty to improve. Let’s discuss some bugs/caveats/issues/problems/future work.

Application policy/promises

Go projects don’t always document whether they need cgo. It’s easy to find out by just trying to build without. But ideally projects would document it as policy. The same applies to being self-contained (with extra files “embed”ed). I’m not a fan of all the “badges” in READMEs. Perhaps we can decide on a word/short sentence for projects to copy?

Semantic versioning for applications

As far as I know, there is no agreed upon meaning of versions for applications. Semantic versioning is aimed at libraries. Go modules are assumed to follow semantic versioning, with major releases are for breaking changes, minor releases for new functionality that’s compatible, and patch releases for bug fixes. How would that apply to applications? Should it be safe to do unattended updates of new minor versions of Go applications? Applications may currently require operator intervention. Major versions may be more important for marketing purposes for applications than for libraries.

Using untested Go toolchains with applications

Automatically updating an application to the latest Go toolchain may mean you’re running an untested build of an application. In my experience Go toolchain patch releases pretty much never break working software. But there’s no guarantee. Ysco currently does wait 1 day by default between discovering an update and installing it. If that would cause breakage, you may hear about it through the grapevine. If the application authors release an updated version with a fix, ysco will currently still continue with its originally scheduled update. It may be better for ysco to skip a version in that situation. A rollout delay longer than 1 day would probably be needed, giving software authors a chance to get a release out and ysco to discover the new version. Of course you can already configure an update delay longer than 1 day.

Scheduling updates

Are there better moments to do updates? Perhaps coordinated with backup schedules? Or looking at how busy the application is at a give moment. Some applications are mostly idle for long parts of the day. Can that be monitored? Ideas welcome.

Zero-downtime updates

Here’s an idea that is not yet implemented: Some applications may be able to exec(2) themselves with a new binary, allowing for zero-downtime updates. If there’s a standardized way to initiate this update (e.g. by sending a signal, like USR1), ysco could be taught to send such a signal. Perhaps there are other mechanisms, like writing a string to a socket. Ideally we would want to check the update succeeded. Ideas welcome.

Ysco can already exec itself to a new version, for updates without downtime. The process ID of the monitored application is passed to the new process. If the update was initiated through an HTTP call, the file descriptor of the HTTP request is passed to the new process too, which writes the HTTP response. Fun stuff.

Gopherwatch.org

For applications that have multiple actively developed minor versions, the gopherwatch DNS will currently only return the latest patch release for the latest minor version. By default, updates to minor releases are not done automatically. So ysco may lose track of new patch releases for older minor versions. The Go module proxy @v/list endpoint does list them all. The gopherwatch DNS endpoint could be changed to return the latest patch releases for all minor releases. Or the request could include the current version, so the latest patch releases of only newer minor versions can be returned. If you know an application maintains multiple minor versions, you can already configure ysco to only look for updates in the Go module proxy.

A related shortcoming is that Go module paths, including the Go module proxy and gopherwatch, require an explicit major version to be present in the path or versions >= 2. New major releases are not currently automatically detected.

Gobuilds.org

Gobuilds is currently in beta. I want to make builds more isolated before taking it out of beta. Builds are currently isolated on linux using bubblewrap (namespaces), but there are still shared files between builds. Ideally, I would do builds in separate VMs, with a warm build cache of the standard library, and with per-OS sandboxing features, like namespaces (and probably a few more techniques) on Linux, jails on FreeBSD, pledge/unveil on OpenBSD. It should also be easier to run your own gobuild instance (help with setting up, automatically managed disk usage), to run it in a verification mode (monitoring the transparency log and verifying all builds), and much more. For transparency, trust and redundancy, a service like this should be run by a small group of people invested in the Go ecosystem. Contact me if you’re interested.

Reproducible builds for frontends

The Go team and ecosystem has put effort into making builds reproducible, without having to run other code than the Go compiler. That’s not easy or possible in most other software ecosystem. Quite a few of my Go applications have a “web frontend”, written in TypeScript. I also commit the JavaScript compilation results so Go can “embed” that code. It’s one of the reasons I write plain TypeScript without frontend frameworks (another is sidestepping frontend dependency churn). It could be an option for a service like gobuilds.org to first build frontend code reproducibly and without running code fetched from dependencies. The frontend ecosystem doesn’t have a single toolchain to do that (it may have many) as far as I know, but please correct me!

Updates through other channels

Wouldn’t it be better to install applications and updates through package managers? I maintain some servers running debian, with unattended updates for automatic security fixes. It works great. But not all applications are packaged, certainly not in all operating systems. And updates don’t always come promptly, or I want to update different applications at different rates. Go is in a nice position with its statically linked, no-dependency binaries. Just copy them to a machine and run them! It’s so easy that getting software packaged is less of a priority. Docker images are similar: self-contained and updated independently of the operating system. For these use-cases, I probably consider it a feature that the “host OS” changes as little as possible. It may not be a good development for operating systems and distributions though. And perhaps also not for the Go applications and their visibility.

Automatic changelogs and git tags

First the background. If mox finds a new version through DNS, it downloads a changelog from a preconfigured URL and, mox being a mail server, composes a message containing the changelog and delivers it to the postmaster mailbox. I’m still pleasantly surprised whenever I get those messages. More applications could benefit from having access to the changelogs: for display in a web interface, or sent as notification. Standardized distribution of changelogs could help, libraries can be developed around it. How could it work? I always add the changelog to annotated git tags. Assuming this is common practice, a new service could be created that polls git repositories for new git tags, fetches their annotations as changelogs, and adds them to its database for future requests. Similar to the Go module proxy.

And one step further, how about adding all git tags and commit hashes (and the hash of the changelog) to a transparency log? It would help detect git tags from being changed (potential supply chain security issue). This is what the Go sum database does for Go modules, but seems useful for all software under version control (not just with git).

Summary

There are situations where managed automated updates, either fully automated or one-click, are a reasonable solution, or at least better than running outdated binaries. Ysco provides such updates for Go binaries. Ysco is easy to run, with little overhead, works for many Go applications unchanged, and is enabled by Go’s excellent support for reproducible cross-compilable builds and module management infrastructure (shoulders of giants, thanks!).

Keep it fresh, run ysco! See https://pkg.go.dev/github.com/mjl-/ysco for documentation.