Accelerating GCCSDK autobuilder using apt-proxy
When developing autobuilder scripts it is inefficient to keep downloading the upstream sources each time, particularly since many packages are large and internet connections may be slow. The following enables the source packages to be downloaded once and kept locally, or on another machine on your network. It only works for those autobuilder packages that use Debian sources as their upstream.
We use the program apt-proxy, which is a specialised web proxy for Debian packages: essentially it builds a local package repository with only those files that have been requested through it.
Debian-based distributions
On Debian-based distributions (including Ubuntu, Mepis, Knoppix, etc) the autobuilder downloads packages using the 'apt' system. From a root shell, install apt-proxy:
debian# apt-get install apt-proxy
apt-proxy is configured with a configuration file in /etc/apt-proxy/ (in Debian sarge it's /etc/apt-proxy/apt-proxy-v2.conf). Notable entries are:
;; Server IP to listen on
address = 127.0.0.1
;; Server port to listen on
port = 9999
;; Cache directory for apt-proxy
cache_dir = /var/cache/apt-proxy
[debian]
;; The main Debian archive
;; Backend servers, in order of preference
backends =
http://ftp.us.debian.org/debian
http://ftp.de.debian.org/debian
http://ftp2.de.debian.org/debian
ftp://ftp.uk.debian.org/debian
So here we create our special webserver on port 9999 and we only allow connections from localhost (this machine) to access it. You may need to change these if you want to store the cache on another machine. The cache will be built up in /var/cache/apt-proxy - this is the default which is created when you install apt-proxy, but if you store the files somewhere else (/scratch/apt-proxy perhaps) create it and give it the aptproxy owner:
debian# mkdir /scratch/apt-proxy debian# chown aptproxy /scratch/apt-proxy
The amount of disc space it'll require depends on how many packages you download: at any time you can delete the contents of the directory and start again if you want to recover some space.
The backends listed are where apt-proxy will go to fetch the files that are requested through it - usually these are the servers for the Linux distribution you're running. Note the name in square brackets - '[debian]' here - this is the name of the directory served by apt-proxy. So apt-proxy with the above configuration will provide http://localhost:9999/debian/. Note you can have more than one directory - another list of backends prefixed by '[ubuntu]' will serve the ubuntu files as http://localhost:9999/ubuntu/, and so on. apt-proxy can cope with as many different distributions as you have disc space for.
So we want to insert apt-proxy between the system and the network. Here's how we do it. In /etc/apt/sources.list there should be a line starting deb-src, something like this:
deb-src http://ftp.uk.debian.org/debian/ unstable main non-free contrib
This specifies where the system gets its packages from. So we should replace it with:
deb-src http://localhost:9999/debian/ unstable main non-free contrib
and then in /etc/apt-proxy/apt-proxy-v2.conf have:
[debian]
backends =
http://ftp.uk.debian.org/debian/
This means apt will ask apt-proxy for its files, and apt-proxy in turn will ask ftp.uk.debian.org. If there's more than one deb-src line listed, it's best to comment all out and just add the one above.
To refresh apt and apt-proxy, we should do:
debian# apt-get update debian# /etc/init.d/apt-proxy restart
Now when the autobuilder fetches its source packages it'll do so via apt-proxy, and not go to the internet if it already has them. This should be much quicker.
Non-Debian distributions
If you use a distribution not based on Debian (RedHat, Fedora, SUSE, etc) you have two problems. Firstly apt probably isn't supported, so the autobuilder won't use it, and secondly apt-proxy might not be packaged. So to start with you'll have to build apt-proxy from source.
Once built and installed, you should set it up as above. You could configure it using one of the Debian backends listed above.
As the system doesn't provide apt, the autobuilder scripts fetch things themselves. To divert them via apt-proxy, edit autobuilder/fetch-program and change:
debarchive=http://ftp.uk.debian.org/debian
to
debarchive=http://localhost:9999/debian
Now the autobuilder scripts should be able to fetch their packages using apt-proxy.
| GCC and GCCSDK pages |
| GCC under RISC OS GCC for RISC OS, GCC tutorial, GCC common switches, GCC for beginners, UnixLib, ELFLoader |
