Accelerating GCCSDK autobuilder using apt-proxy: Difference between revisions

From RISC OS
Jump to navigationJump to search
(Start page)
 
(Add more)
Line 1: Line 1:
When developing autobuilder scripts it is inefficient to keep downloading the source packages each time, particularly since many packages are large and internet connections may be slow. The following enables the source packages to be downloaded once and kept locally, or on another machine on your network. It only works for those autobuilder packages that use Debian sources as their upstream.
When developing autobuilder scripts it is inefficient to keep downloading the upstream sources each time, particularly since many packages are large and internet connections may be slow. The following enables the source packages to be downloaded once and kept locally, or on another machine on your network. It only works for those autobuilder packages that use Debian sources as their upstream.


We use the program [[http://apt-proxy.sourceforge.net/ apt-proxy]], which is a specialised web proxy for Debian packages: essentially it builds a local package repository with only those files that have been requested through it.
We use the program [http://apt-proxy.sourceforge.net/ apt-proxy], which is a specialised web proxy for Debian packages: essentially it builds a local package repository with only those files that have been requested through it.


==Debian-based distributions==
==Debian-based distributions==


On Debian-based distributions (including Ubuntu, Mepis, Knoppix, etc) the autobuilder downloads packages using the 'apt' program. From a root shell, install apt-proxy:
On Debian-based distributions (including Ubuntu, Mepis, Knoppix, etc) the autobuilder downloads packages using the 'apt' system. From a root shell, install apt-proxy:
debian# apt-get install apt-proxy
debian# apt-get install apt-proxy


Line 13: Line 13:
;; Server IP to listen on
;; Server IP to listen on
address = 127.0.0.1
address = 127.0.0.1

;; Server port to listen on
;; Server port to listen on
port = 9999
port = 9999

;; Cache directory for apt-proxy
;; Cache directory for apt-proxy
cache_dir = /var/cache/apt-proxy
cache_dir = /var/cache/apt-proxy

[debian]
[debian]
;; The main Debian archive
;; The main Debian archive
Line 29: Line 29:
ftp://ftp.uk.debian.org/debian
ftp://ftp.uk.debian.org/debian


So here we create our special proxy on port 9999 and we only allow connections from localhost (this machine) to access it. You may need to change these if you want to store the cache on another machine. The cache will be built up in /var/cache/apt-proxy - create this directory and give it the aptproxy owner:
So here we create our special webserver on port 9999 and we only allow connections from localhost (this machine) to access it. You may need to change these if you want to store the cache on another machine. The cache will be built up in /var/cache/apt-proxy - this is the default which is created when you install apt-proxy, but if you store the files somewhere else (/scratch/apt-proxy perhaps) create it and give it the aptproxy owner:
debian# mkdir /var/cache/apt-proxy
debian# mkdir /scratch/apt-proxy
debian# chown aptproxy /var/cache/apt-proxy
debian# chown aptproxy /scratch/apt-proxy
The amount of disc space it'll require depends on how many packages you download: at any time you can delete the contents of the directory and start again if you want to recover some space.
The amount of disc space it'll require depends on how many packages you download: at any time you can delete the contents of the directory and start again if you want to recover some space.

The backends listed are where apt-proxy will go to fetch the files that are requested through it - usually these are the servers for the Linux distribution you're running. Note the name in square brackets - '[debian]' here - this is the name of the directory served by apt-proxy. So apt-proxy with the above configuration will provide http://localhost:9999/debian/. Note you can have more than one directory - another list of backends prefixed by '[ubuntu]' will serve the ubuntu files as http://localhost:9999/ubuntu/, and so on. apt-proxy can cope with as many different distributions as you have disc space for.

Revision as of 01:59, 15 February 2007

When developing autobuilder scripts it is inefficient to keep downloading the upstream sources each time, particularly since many packages are large and internet connections may be slow. The following enables the source packages to be downloaded once and kept locally, or on another machine on your network. It only works for those autobuilder packages that use Debian sources as their upstream.

We use the program apt-proxy, which is a specialised web proxy for Debian packages: essentially it builds a local package repository with only those files that have been requested through it.

Debian-based distributions

On Debian-based distributions (including Ubuntu, Mepis, Knoppix, etc) the autobuilder downloads packages using the 'apt' system. From a root shell, install apt-proxy: 

debian# apt-get install apt-proxy

apt-proxy is configured with a configuration file in /etc/apt-proxy/ (in Debian sarge it's /etc/apt-proxy/apt-proxy-v2.conf). Notable entries are: 

;; Server IP to listen on
address = 127.0.0.1

;; Server port to listen on
port = 9999

;; Cache directory for apt-proxy
cache_dir = /var/cache/apt-proxy

[debian]
;; The main Debian archive
;; Backend servers, in order of preference
backends = 
        http://ftp.us.debian.org/debian
        http://ftp.de.debian.org/debian
        http://ftp2.de.debian.org/debian
        ftp://ftp.uk.debian.org/debian

So here we create our special webserver on port 9999 and we only allow connections from localhost (this machine) to access it. You may need to change these if you want to store the cache on another machine. The cache will be built up in /var/cache/apt-proxy - this is the default which is created when you install apt-proxy, but if you store the files somewhere else (/scratch/apt-proxy perhaps) create it and give it the aptproxy owner: 

debian# mkdir /scratch/apt-proxy
debian# chown aptproxy /scratch/apt-proxy

The amount of disc space it'll require depends on how many packages you download: at any time you can delete the contents of the directory and start again if you want to recover some space.

The backends listed are where apt-proxy will go to fetch the files that are requested through it - usually these are the servers for the Linux distribution you're running. Note the name in square brackets - '[debian]' here - this is the name of the directory served by apt-proxy. So apt-proxy with the above configuration will provide http://localhost:9999/debian/. Note you can have more than one directory - another list of backends prefixed by '[ubuntu]' will serve the ubuntu files as http://localhost:9999/ubuntu/, and so on. apt-proxy can cope with as many different distributions as you have disc space for.

Retrieved from "http://www.riscos.info/index.php?title=Accelerating_GCCSDK_autobuilder_using_apt-proxy&oldid=1777"