Accelerating GCCSDK autobuilder using apt-proxy

When developing autobuilder scripts it is inefficient to keep downloading the upstream sources each time, particularly since many packages are large and internet connections may be slow. The following enables the source packages to be downloaded once and kept locally, or on another machine on your network. It only works for those autobuilder packages that use Debian sources as their upstream.

We use the program apt-proxy, which is a specialised web proxy for Debian packages: essentially it builds a local package repository with only those files that have been requested through it.

Debian-based distributions

On Debian-based distributions (including Ubuntu, Mepis, Knoppix, etc) the autobuilder downloads packages using the 'apt' system. From a root shell, install apt-proxy:

debian# apt-get install apt-proxy

apt-proxy is configured with a configuration file in /etc/apt-proxy/ (in Debian sarge it's /etc/apt-proxy/apt-proxy-v2.conf). Notable entries are:

;; Server IP to listen on
address = 127.0.0.1

;; Server port to listen on
port = 9999

;; Cache directory for apt-proxy
cache_dir = /var/cache/apt-proxy

[debian]
;; The main Debian archive
;; Backend servers, in order of preference
backends = 
        http://ftp.us.debian.org/debian
        http://ftp.de.debian.org/debian
        http://ftp2.de.debian.org/debian
        ftp://ftp.uk.debian.org/debian

So here we create our special webserver on port 9999 and we only allow connections from localhost (this machine) to access it. You may need to change these if you want to store the cache on another machine. The cache will be built up in /var/cache/apt-proxy - this is the default which is created when you install apt-proxy, but if you store the files somewhere else (/scratch/apt-proxy perhaps) create it and give it the aptproxy owner:

debian# mkdir /scratch/apt-proxy
debian# chown aptproxy /scratch/apt-proxy

The amount of disc space it'll require depends on how many packages you download: at any time you can delete the contents of the directory and start again if you want to recover some space.

The backends listed are where apt-proxy will go to fetch the files that are requested through it - usually these are the servers for the Linux distribution you're running. Note the name in square brackets - '[debian]' here - this is the name of the directory served by apt-proxy. So apt-proxy with the above configuration will provide http://localhost:9999/debian/. Note you can have more than one directory - another list of backends prefixed by '[ubuntu]' will serve the ubuntu files as http://localhost:9999/ubuntu/, and so on. apt-proxy can cope with as many different distributions as you have disc space for.

So we want to insert apt-proxy between the system and the network. Here's how we do it. In /etc/apt/sources.list there should be a line starting deb-src, something like this:

deb-src http://ftp.uk.debian.org/debian/ unstable main non-free contrib

This specifies where the system gets its packages from. So we should replace it with:

deb-src http://localhost:9999/debian/ unstable main non-free contrib

and then in /etc/apt-proxy/apt-proxy-v2.conf have:

[debian]
backends =
         http://ftp.uk.debian.org/debian/

This means apt will ask apt-proxy for its files, and apt-proxy in turn will ask ftp.uk.debian.org. If there's more than one deb-src line listed, it's best to comment all out and just add the one above.

To refresh apt and apt-proxy, we should do:

debian# apt-get update
debian# /etc/init.d/apt-proxy restart

Now when the autobuilder fetches its source packages it'll do so via apt-proxy, and not go to the internet if it already has them. This should be much quicker.

Non-Debian distributions

If you use a distribution not based on Debian (RedHat, Fedora, SUSE, etc) you have two problems. Firstly apt probably isn't supported, so the autobuilder won't use it, and secondly apt-proxy might not be packaged. So to start with you'll have to build apt-proxy from source.

Once built and installed, you should set it up as above. You could configure it using one of the Debian backends listed above.

As the system doesn't provide apt, the autobuilder scripts fetch things themselves. To divert them via apt-proxy, edit autobuilder/fetch-program and change:

    debarchive=http://ftp.uk.debian.org/debian

to

    debarchive=http://localhost:9999/debian

Now the autobuilder scripts should be able to fetch their packages using apt-proxy.