[Rpcemu] Patch for > 2G files in HostFS

Matthew Howkins rpcemu-list at howkins.me.uk
Sun Oct 23 10:40:35 PDT 2011


> Ah, that's a frequently called out myth.
> In PRM 5a (in the section 'FileCore') the overview section describes the
> limitations present, at RISC OS 3.70 at least. To paraphrase:
>
>  Filecore = max file size 2^31 -1
>  FileSwitch = max file size 2^32 -1
>
> as HostFS is a FileSwitch client it should be able to support 4G files as
> far back as RISC OS 3.70. My recent fixes to the RISC OS 5 sources were, I
> think without exception (and without going back and looking!), due to the 32
> bitting process meaning that functions that previously preserved the
> overflow bit now accidentally set it when comparing big files. This issue is
> quite distinct from FileCore being stuck at 2G.

I have given your proposed changes some serious thought, and done some testing.

I was concerned that enabling support for files up to 4GB would expose the
assumption that files are always less than 2GB in size, and this would manifest
itself in bugs in RISC OS and its applications.

You stated that RISC OS 3.70 and later support files up to 4GB, so I performed
my testing with both RISC OS 4.02 and RISC OS 5. I used a version of HostFS
that fully supported large files (on 64-bit Linux this support is enabled as an
inherent feature of a 64-bit executable).

My first test was to copy a 3GB file to a new file using the Filer. This
immediately failed on RISC OS 4.02 with an error 'No readable memory at this
address'.

A glance at the source code of RISC OS 5 shows that the FilerAction code uses
a signed 32-bit integer to keep track of file sizes and copy progress, though I
see that you have very recently modified the code on riscosopen.org to use an
unsigned 32-bit integer. I will assume that all other versions of RISC OS
contain the signed variant - this seems reasonable based on my testing. This
therefore has an implicit assumption that the file will always be less than 2GB
in size, and never considers the possibility of a larger file, and has
inadequate error handling in that case.

HostFS is regularly used with RISC OS versions from 3 to 6. It needs to work
with all of these, and must conform to their assumptions and limitations. For
reasons of compatibility the default behaviour of HostFS must be limited to a
2GB file size.

RPCEmu is also used to run a lot of old/legacy applications. Those which are
coded in C and using the standard C API will not support files >=2GB, and if
they are presented with files of this size will probably fail in
unpredictable ways.

Given the constraints listed above, I am not willing to accept your changes as
the support for up to 4GB files simply will not work for the vast majority of
RPCEmu users.

When RISC OS gains full support for 64-bit file sizes through the use of a new
API for filesystems to use, I am keen to modify HostFS to support this feature,
and will modify it at that time.

I acknowledge that the current behaviour of HostFS when faced with large files
could best be described as 'undefined' - it varies with platform and whether it
is compiled for 32-bit or 64-bit. I will work to improve this, but intend to
retain the widest possible compatibility.

Matthew

PS My apologies to Sprow who will receive this twice - I meant my
reply to go the list, not to him alone.



More information about the Rpcemu mailing list