[MLton-user] How to access large amounts of data through FFI

[Vesa Karvonen]

On Fri, Oct 10, 2008 at 9:24 PM, Ville Tuulos <tuulos at gmail.com> wrote:
> I would like to access the data in Mlton without copying it first, due
> to obvious performance reasons. The data itself is a flat array of
> native 32-bit ints. Since it's an immutable array of native ints, I
> was hoping that I could access it as an int vector (or something
> similar) in Mlton.
>
> How to construct and pass a valid int vector from C to SML through
> Mlton's FFI, preferably without copying the data first?

Unfortunately, you can't access a mmapped region of memory as an
ordinary SML vector, because ordinary SML vectors live in MLton's
garbage collected heap (and have a special header for MLton's runtime
and may be moved by MLton's GC).

> The FFI interface seems pretty straighforward but I couldn't find any
> documentation how arrays and vectors should be initialized based on a
> C pointer and size of the data. Or, if the int vector is not a good
> idea, what would be a better way to get access to the memory space?

Like suggested by Florian Weimer, likely the most efficient approach
is to access the data using MLton.Pointer.

The attached files contain an example of a way to do it.  The example
contains the beginnings of a RawVector module that allows one to mmap
a file and access it similarly to an ordinary vector.  As an example,
it opens a file ("data"), prints the length of the resulting RawVector
and then prints all the elements of the RawVector.

For (my) convenience, the example makes (rather minimal) use of an
extended basis library from the "mltonlib" repository.  You can check
out the mltonlib repository with the following command:

  svn co svn://mlton.org/mltonlib/trunk mltonlib

You'll then need to modify the Build.sh script to point to the
location of mltonlib in order to build the example with the script.

-Vesa Karvonen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Build.sh
Type: application/x-sh
Size: 254 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/3d0800ed/Build.sh
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data
Type: application/octet-stream
Size: 8 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/3d0800ed/data.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap.c
Type: text/x-csrc
Size: 554 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/3d0800ed/mmap.c
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap.mlb
Type: application/octet-stream
Size: 226 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/3d0800ed/mmap.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap.sml
Type: application/octet-stream
Size: 2273 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/3d0800ed/mmap-0001.obj