Gitian Build Process

Intro

In the first of my write-ups following the ReddHead newsletter for December, I wanted to step through the process of how the building of binary executables that you run as a wallet occurs.

Like everything in this Cryptocoin space, there are a number of ways that generating the executable binary can be achieved depending on the experience and expertise of the end user

Building from source

Originally with Bitcoin, the process of building and distributing the wallet was purely on the shoulders of the end user.
They would need to download the source files from Github, and setup their build environment on their PC, and then compile the source into an executable.

For anyone who has done this, there are many pitfalls particularly since the compile process required several external libraries (OpenSSL, miniupnp, Berkley DB, Boost). If you install the wrong version, you could at worst fail to compile, or use the wrong library, or introduce untested libraries. As anyone could contest, sometimes the process goes smoothly, and sometimes it goes off the rails. Generally speaking, for anyone who does decide to compile from source, it is a good learning experience for the inner working of what goes into building the wallet

In fairness, the documentation has improved and for the most part with a bit of technical knowledge you could build the wallet yourself but be prepared to do some research also.

For the distributed binaries for the general public, this would be performed by one or more developers. Following the guidelines of building from source, this would result in the required executable binaries. With a bit of luck, they would provide a SHA256 of the output so that when you download the file, you could compare that you have the same file as was originally generated.

The Gitian Build Process

The Gitian build process takes the build process to a whole new level.
From the website (https://gitian.org) Gitian is a secure source-control oriented software distribution method. This means you can download trusted binaries that are verified by multiple builders
Gitian uses a deterministic build process to allow multiple builders to create identical binaries. This allows multiple parties to sign the resulting binaries, guaranteeing that the binaries and tool chain were not tampered with and that the same source was used. It removes the build and distribution process as a single point of failure.

So that’s the high level view
Underneath, what this means is that there is a described process to create a consistent build environment for Windows/Linux/Osx that will result in binaries built in a consistent manner.

Getting going

My involvement with this has been quite a learning experience.
Locally I run a windows laptop. Laziness has probably kept me from moving to a Linux setup, but that is neither here nor there for this topic. Getting Gitian up and running is fairly well described as it relies on running Virtual Machines in any case. In fact the process results in a virtual machine, running on a virtual machine.

There are a couple of ways to go about getting setup and depending on which path you go it is fairly straight forward. The reason it took me some time to get it going was determined by the selection process along the way, and then needing to back out or take alternate route. Although a little tedious, it allowed me to become familiar with the process and the advantage or limitations
For example, VirtualBox is a Virtual Machine environment that you may be familiar with, and most of the documentation supports this. However, I found that doesn’t really support all scenarios of running a virtual machine within a virtual machine. Whats required is that your environment supports Intel VT-x or AMD-V
I found that VirtualBox didn’t necessarily pass the cpu virtualisation through to the guest while VMWare does support this virtualisation. So in any case I was able to do an alternate path of test and still arrive at the same results.

The basic setup requires you create a Virtual Machine running either Debian or Ubuntu in a very barebones configuration (No GUI required) with terminal access.
Onto this virtual machine, you install the Gitian Builder package, clone your Reddcoin source repo, and download the dependency libraries for building the wallet.

The gitian builder will invoke creating an additional virtual machine (KVM or LVM) based on your selection (nested VM)
The first step is to create those virtual machines using the gitian tools
Then the dependency packages that will be used to generate the final binaries. These are created for Windows, Linux and Osx
You compile your binary files for Windows, Linux, and Osx (and in 32 & 64bit), again using the Gitian Builder
The resulting files are then signed using your GPG key.
I was just testing this myself, so I didn’t complete the next step, but in essence once 2 or more builds have been created, you can compare the sigs, and then prepare to package the installers for each environment.

Conclusion

Suffice to say, initially it wasn’t exactly a straightforward process. There were a number of situations that prevented a successful build. However, after overcoming them I am now successfully building some of my changes for testing, I am confident that we continue to have a solid build process.

I have glossed over some of the finer details, but if you feel you would like me to expand, please comment and I will add it in.

Gnasher How are you able to assure that you are not building in a bug or a fault that wont be obvious until much much later…ie…a counting or rounding error? How does the code compile after you build a module…or module…do you get a check sum or some other indication that what you have written is correct?

Hey Redshift
The purpose if going through the Gitian build is not to assure that bugs are not included.
The purpose is to have a trusted system where multiple people create identical binaries.
If there is a bug in the code and it has been overlooked, then in theory everyone would build the same bug (and it would be included in the final build) and all builds would be identical.

Bugs can only be eliminated by vigourous testing before committing to final.

The real purpose of this process is to attempt to provide a trusted build output…
That is, not relying on a single person to build process