Experiences with Go modules
Go 1.11beta2 is out. And it includes the new Go modules support, what started as vgo. Go beta’s are delightfully easy to try out:
go get golang.org/dl/go1.11beta2
go1.11beta2 download
I wanted to use it. See how it feels. And… it works pretty good! I even started using the new go modules to manage my vendor directories.
Below is my description of how I started using Go modules in a project I’m currently working on (closed source, can’t show it).
initialize
First, you want to initialize modules in your repo:
go1.11beta2 mod -init
ERROR!!!
go: modules disabled inside GOPATH/src by GO111MODULE=auto; see 'go help modules'
My repo – and probably yours – lives in my $GOPATH. To use the new Go modules, you need to enable modules for use within $GOPATH. It’s all explained by the error message, so no problem. So you do:
export GO111MODULE=on
go1.11beta2 mod -init
That worked.
Then I ran a regular go build. Packages were fetched, and code was compiled. That was easy! Luckily, the dependencies I was using had reasonable release tags.
go.mod and go.sum
You get these new files.
I opened the go.mod file, and the packages looked familiar from my vendor/ directory. Versions looked OK too. I had read rsc’s vgo blog posts before, so the file looked familiar.
Then the go.sum file. What to do with it? Should you add it to your repositories? Yes, it seems so. Go.sum contains hashes of the downloaded package versions. It is automatically updated when you fetch newer package versions. When you fetch packages from the internet, it will verify those sums. So keeping the file in the repo ensures anyone fetching those dependencies (including a future you) can verify they get the same code. Definitely desirable. It seems go.sum will automatically be purged of package versions no longer used #26381.
vendor
Go1.11beta2 is already useful for managing your vendor directory! Until now I was managing vendored libraries manually. I would set $GOPATH to some place, run go get, and move the fetched packages to vendor/. It worked, but required too much effort. Now with Go modules I can just run:
go1.11beta2 mod -vendor
And my vendor directory will be updated. It’s easy and quick, and has another upside. My vendor directory now has far fewer files. That’s because go mod excludes test code, and excludes directories that aren’t needed when building the package. Lots of clutter removed from vendor/!
This also has a downside though. I was using one of those now-excluded directories as part of my build script. To get to the point: I have a Makefile and it would compile and run vendor/bitbucket.org/mjl/sherpa/cmd/sherpadoc/. Sherpadoc is a tool that parses my Go code and turns Go comments into documentation for a web API my project exports. Just like godoc. The documentation is written to a file and compiled into the program so it is available at runtime. Unfortunately, go mod is so lean and mean that it knows that code isn’t needed, and removes it.
How to get it back into my vendor directory? I couldn’t find an easy way. You cannot just add the needed full import path to the go.mod dependencies: Again “go mod” is smart enough to see you’re still not using it, so it will remove the dependency. You also cannot import the cmd/sherpadoc code in a .go file in your project: You are not allowed to import a “main” package in a Go program.
In the end, I did manage to include the sherpadoc code. Not very elegant, but it works: I modified the bitbucket.org/mjl/sherpa repository (it’s mine). I moved the Go code for the command cmd/sherpadoc to a subpackage bitbucket.org/mjl/sherpa/sherpadoc, with the old main() renamed to the exported Main(). I wrote a new trivial cmd/sherpadoc/main.go with just an import of the new subpackage and a main() that calls said Main(). Then in my own project I added the same trivial cmd/sherpadoc/main.go. Now go mod -vendor sees the sherpadoc “library” code is used, and keeps it in the vendor-directory. I also had to update my Makefile to build and run the new cmd/sherpadoc instead of the previous vendor/bitbucket.org/mjl/sherpa/cmd/sherpadoc.
If there is a way to “import” (and not use) a “main” package, I’m interested in hearing it. That would provide a simpler solution with fewer changes.
OK, this was a relatively quick fix. I didn’t want to manually keep track of changes to the sherpadoc code. But the approach only worked because I can change the sherpa repository. For externally-managed repos I would have to fork. If someone sent me a pull request with the change I made, I wouldn’t accept happily. There’s also the downside of an extra trivial Go file in my project’s repository. But that’s minor, and already offset by the shorter commands in the Makefile. (;
After a go mod -vendor, you’ll also get a vendor/modules.txt. I couldn’t find official docs on that file. It look it’s just a log of what was synced into vendor/. I’m also committing it to my repo’s.
practice
The good thing about running into a snag like the above is that you get to practice the “go mod” subcommands (btw, they are disguised as flags, but act like commands).
I’ve used go mod -replace=$oldimport=$localdir. One caveat is that you need an absolute local path for $localdir. I tried using a local relative path, but it wasn’t accepted. Probably because go has no way to telling the difference between a local directory and a package name, they look the same. Easily fixed by using $PWD/$localdir.
I also used go mod -require. I needed to upgrade to a later, not yet tagged, version of a dependency. With my previous strategy of go get on a temporary and empty $GOPATH, I would get the latest master commits. But go mod defaults to the latest tagged releases. Good idea, but it results in different code. Using go mod -require=$package@$commithash did the trick, of course with the two variables filled in.
conclusion
Go modules are the good stuff. You can already use go1.11beta2 to manage your dependencies. Of course while still building your releases with the stable Go compilers. But if your builds are more than a simple go build, you might run into similar trouble with useful vendored Go code “helpfully” purged.