xcode license nonsense wasting time

A good chunk of the day went into debugging hanging builds in Ding, a new build server. Turns out it wasn’t dings fault.

I had just added privilege separation to Ding. Unfortunately, builds were now hanging… I checked and double check the code. Tested different configuration. Tested the new code on OpenBSD and Linux. And… the hangs only happened on macOS! And only with the privsep mode on, where there is a root process that starts the builds under a unique uid, and passes the stdout & stderr file descriptors to the unprivileged webserver processes. If I turned that mode off. All processes are created and communication is done with the same flow and code, just not with different uids, and the builds would not hang. Weird?

So macOS must be at fault. Initially I thought it must be related to the new privsep code that passes file descriptors between processes. I thought the “hanging” was because reads on the file descriptors from the build command were hanging. The build commands were started, I could see them in ps, running. I thought the output of lsof showed the pipes between the processes were filled up. I now know I was wrong.

After quite a bit of testing, I found the real cause. It was actually go build that as hanging. No progress at all. Why? Because it calls xcodebuild -license check. And that was hanging. Yes… for some reason compiling Go code requires running xcodebuild. And for some reason, running xcodebuild -license check will hang if you run it under a uid that doesn’t have a regular system account. That explains why running without the new privsep mode wouldn’t hang the builds.

In case you know the cause, let me know. The test:

$ xcodebuild -version
Xcode 9.1
Build version 9B55

$ sudo -u '#10000' xcodebuild -license check
# hangs indefinitely

Now create a user with that uid, and try again:

sudo sysadminctl -addUser testuser -UID 10000

Let me know if you have a solution or an explanation.