Autobuilder Development and Packaging

Information on using the Autobuilder is listed on the GCCSDK Autobuilder page. This page is about adding packages to it.

The specific aims of an Autobuilder package are:

• Supply information on where to get the sources
• Perform any special build steps, especially for cross compiling
• Generate a RiscPkg package suitable for deployment

Given minimal information, the Autobuilder knows how to do almost all of these steps by itself most of the time. But some packages deviate, and explicit instructions need to be given.

Executive summary

The 11 step guide to porting a new program:

1. Install and build GCCSDK (ideally on a Linux machine, as that's the best tested. If you don't have a Linux machine, try a minimal Debian in a virtual machine)
2. Make a directory autobuilder/category/packagename
3. Copy autobuilder/setvars-template to autobuilder/category/packagename/setvars
4. Edit setvars as appropriate (including adding a pointer to the source code if it isn't available as a Debian package)
5. Type ./build -d packagename
6. Fix things that didn't work (look in autobuilder/category/packagename/last-failure). Perhaps add extra patch files as eg autobuilder/category/packagename/patchfile.p or build other things first by adding a 'depends' file. May also need adding extra functions to UnixLib or porting other libraries.
7. Test your program! A network share to a RISC OS machine is handy for this, or use RPCEmu
8. Add an ab_package() section to setvars to package it appropriately
9. Submit your changes to GCCSDK SVN (eg mail patches to the mailing list). Don't forget any changes to UnixLib or the build scripts if you needed them.
10. Someone will build it and upload it to the website
11. When you get stuck, contact the GCCSDK mailing list

Anatomy of an Autobuilder package

Note: some packages follow older conventions, and do not look exactly like what is described here. These are in the process of being updated.

To start, download a copy of the Autobuilder from SVN (in case you don't have it already as part of the gccsdk checkout):

$ svn co svn://svn.riscos.info/gccsdk/trunk/ gccsdk
$ cd gccsdk/autobuilder

We'll look as an example in the directory cli/tar. This directory contain the necessary specifics to build GNU tar program with Autobuilder. The heart of an Autobuilder package is a file called setvars. Apart from the ab_package function discussed below tar's only has:

export gl_cv_sys_struct_timeval=yes
export gl_cv_func_gettimeofday_clobber=no

Normally the Autobuilder will make a best guess at how to build a package, so all that's needed are to specify in those areas it gets it wrong. The main work is done by the fetch* and build* collections of scripts in the top level of the Autobuilder tree. As well as setting various key variables and providing some functions, the setvars script is run just before any configuration script so you can use it to fix up anything that the automatic steps missed.

In the example above, tar uses a standard autoconf layout, its sources are available in Debian and holds no particular surprises, so the build description is very short. If it was fetched from some unusual play, it might use the AB_URL variable. The two variables set here are instructions to autoconf about the handling of certain RISC OS functions, that it is unable to guess for itself when cross compiling. The Autobuilder also sets many more system-wide variables to make autoconf work correctly in most cases without additional instruction.

setvars variables

If no URL is given the Autobuilder will try to fetch the package of the same name from Debian's 'unstable' repository. Here we don't want that, so we explicitly give a URL in AB_URL of where the sources can be found. This can be a .tar.gz, a .tar.bz2 or a .zip file. Alternatively you can specify a CVS or SVN repository. For example, browser/firefox2/setvars has:

AB_CVS_ROOT=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
AB_CVS_TAGORBRANCH=FIREFOX_2_0_0_14_RELEASE
AB_CVS_MODULE=mozilla/client.mk

while audio/tremor/setvars has:

AB_SVN=http://svn.xiph.org/trunk/Tremor

All possible servars variables are listed, together with a short description, in the setvars-template file.

Patch files

Files ending .p are patches which the Autobuilder will apply to the downloaded source tree. You can do this by:

$ cd <root of package's sources>
$ cp <subdir>/file_to_be_patched.c <subdir>/file_to_be_patched.c.orig
$ [ Edit <subdir>/file_to_be_patched.c, build again, make sure your changes are minimal and needed and have the desired effect ]
$ diff -u <subdir>/file_to_be_patched.c.orig <subdir>/file_to_be_patched.c > <...>/autobuilder/<package subdir>/<package name>/file_to_be_patched.c.p
$ svn add <...>/autobuilder/<package subdir>/<package name>/file_to_be_patched.c.p

For this to work the file naming in the first two lines of the patch file must be correct - ensure you just have a relative path starting at the top level of the unpacked source tree. You need to see something like for the first two lines:

 --- src/tar.c.orig	2009-12-03 11:00:06.000000000 -0800
 +++ src/tar.c	2009-12-03 11:01:00.000000000 -0800

Dependencies

Packages may depend on other packages. Normally this uses the Debian dependency information, but you may specify packages manually using a depends file. This just has a list of package names - for an example see browser/firefox2/depends

Building the package and fixing problems

Once you have a setvars file, try

cd autobuilder
./build -d packagename

to build the package. It probably won't build for some reason or other. Your job is to fix this. You'll find the build tree in autobuilder/packagename (-d means keep it around even if the build succeeded), including any output files. The log of what happened will be in autobuilder/category/packagename/last-failure (or last-success, or last-depends).

Common build failures:

• It might need some library that's in the autobuilder you haven't built yet (add the name of the library's autobuilder directory to a 'depends' file next to 'setvars' and rebuild - it'll force the library to build also).
• It might need you to port a library first (so create a package for the library by the same method and make sure that builds).
• It might need some function in UnixLib, or that you need to write yourself
• It might fail to configure - usually because ./configure makes and tries to execute a RISC OS binary on your Linux system. The simplest way to resolve this is to find out what you need to set (look in the ./configure file and config.log) - for example if it tests for the strerror() function ./configure will set ac_cv_func_strerror=yes as UnixLib has it. So add 'export ac_cv_func_strerror=yes' to the ro-config script to make it skip this test (don't forget to submit this change to SVN when you've got everything working). If ./configure compiles a test program to find something out, you might need to compile that program and take it to a RISC OS machine to find out the answer.
• It might need some tool to run as part of the build process. You may have to install additional Linux packages on your machine. Use sites like packages.debian.org (or the alternative for your distro) to find out which packages contain that tool.
• It may require modifications to the source to fix problems - make files ending in .p in the package directory, and they'll be automatically applied as part of the build
• It might be missing symbols - is there a library you need to install first? Do you need to add a -llibsomething to a makefile somewhere?(Linux may pull in some libraries that you need to specify explicitly in a RISC OS build)
• It might need explicitly setting static (or dynamic) linking, rather than assuming the default (usually an option to ./configure, so add it to the AB_CONFLAGS line in setvars)
• It might make assumptions that don't hold - it might build, but crash. The classic one is structure alignment which is different between ARM and x86 - have a look at 'packed' structs. Also filing system differences (dots and slashes) can cause problems - UnixLib has filename translation but you may need to set the relevant option (especially if you want to accept RISC OS filenames).

Sometimes build error messages can be cryptic - it's always worth googling to see if other people have had the same problems (often the problems RISC OS has are also faced by people building on non-Linux platforms like MacOS X or FreeBSD)

Rarely, you need to do something to the source code after it is unpacked but before 'configure' is run. This can go in a script called 'preprocess' next to 'setvars'.

Packaging your program

setvars functions

When producing a Zip package, the ab_package shell function is called to choose which files are to be placed in the zipfile.

There are three main types of packages - the first type contains a standard RISC OS application, the second is a container for command line components, and the third contains development libraries and headers. In future, there will be another type generated along with the library one to contain a shared library.

RISC OS Application

XJig is a typical game, and its packaging looks like this:

ab_package() {
  ab_create_app XJig Apps/Games
  ab_makerun $S/xjig$AB_EXEEXT

  cp -av $S/tina.gif $A/
  rman -f HTML $S/xjig.man > $A/\!Help,faf

  $AB_HOME/add-riscpkg -unixlib -depends Tinct
}

In the final package, the path Apps/Games/!XJig will be created, which is part of the RiscPkg structure. The !XJig application is copied from autobuilder/games/xjig/!Xjig by the ab_create_app command.

The ab_makerun copies the actual game binary, but also importantly, it converts it from a static ELF binary (it will complain if it is a dynamic ELF binary) to a RISC OS AIF, and will also calculate the required WimpSlot for the application. This will be substituted for 'WIMPSLOT' in the !Run file.

The rest of the commands copy required files, and then perform the final packaging steps.

Command Line Applications

For wget, which is a command line application:

ab_package() {
  ab_create_command_app Wget Apps/Network
  ab_add_commands $S/src/*$AB_EXEEXT

  rman -f HTML $S/doc/wget.info > $A/\!Help,faf

  cp -av $S/README $A
  cp -av $S/AUTHORS $S/COPYING $S/MAILING-LIST $A
  cp -av $S/NEWS $S/ChangeLog $A

  $AB_HOME/add-riscpkg -unixlib
}

In this case, a !Wget application will be created. If !Wget already exists in the autobuilder directory, that will be copied. ab_add_commands is used in this case to add any command line commands. This will generate Alias variables in the !Boot file.

Note the variables in use in these scripts:

$S - source directory. e.g, the directory from which 'make' was run.
$D - destination, top level of packaging.  This is often not used.
$H - home, or directory containing setvars for the package
$A - application directory created after call to ab_create_command_app
$AA - actual app name, in this case 'Wget'.  Normally not used by setvars.

Libraries

To be added.

RiscPkg data

The information in the RiscPkg control file is mostly generated from the flags in the setvars file (for example AB_PVERSION and AB_MAINTAINER).

Sometimes you aren't packaging a Debian package and want to supply a description for the RiscPkg archive. Create a RiscPkg directory next to setvars, and add a Control file that just has a Description entry, formatted as the RiscPkg policy. For example:

Description: An example program
 This program does things.  They are not useful things.  But it is
 merely as an example to show that such is possible.
 .
 I don't recommend installing it.

Note the space before each line of the long description, and paragraph breaks marked by dots. This will be appended to the automatically generated information in the control file.

In the same place you can also add a Copyright file with the licence conditions, which will be include as the RiscPkg copyright file. If you don't a standard autobuilder Copyright file will be generated.

Note that AB_LICENCE=Free means the software follows the Open Source Definition, not just that it's zero-cost. Any other software must be Non-free.

zipping up

The autobuilder will automatically unzip source and compile your package, so you shouldn't need to zip it up yourself. But in case you need them, there are RISC OS filetype-preserving zip and unzip tools in the GCCSDK env/ directory. You need to use env/zip -, and env/unzip -F to enable the filetype behaviour. This comes in handy if the source code was intended for RISC OS in the first place.