[MLton-user] How to access large amounts of data through FFI

[Vesa Karvonen]

On Sat, Oct 11, 2008 at 7:49 AM, Ville Tuulos <tuulos at gmail.com> wrote:
[...]
> Actually when I got the Florian's reply about MLton.Pointer, I started
> to consider writing something like RawVector which would support map,
> foldl etc. over mmaped memory. Going through your code will be even
> better exercise for me.

I extended my example somewhat.  The most significant change in the
attached version is that it uses MLton's finalizers
(http://mlton.org/MLtonFinalizable) to munmap the mapped region
automatically when the RawVector is garbage collected.  This can be
convenient, but assuming that the mapped file is large, it is usually
best to free (munmap) it explicitly (which is also supported), because
it might take a long time before the finalizers are run.  The use of
finalizers also adds some "access overhead" to the RawVector
implementation and also has some other effects.  For something like
the sub operation the access overhead may be significant.  Aggregate
operations (foldl, find, app, ...) only incur the access overhead
once.

Implementing map, i.e. a function of type ('a -> 'b) -> 'a t -> 'b  t,
for something like RawVector is not entirely straightforward.  The
main problem is that if one is using it to map very large files to
memory, like suggested in your initial message, then creating a new
RawVector by mapping a function over it is likely to be inefficient.
Rather than use map, it is probably better to use "lower level"
functions like foldl directly or use lazy sequence of some form.  One
possibility would be to use "iterator combinators" or something
similar that allows you to lazily map over the raw vector.  Iterator
combinators are explained on the http://mlton.org/ForLoops page (note
that RawVector.for is essentially an iterator function and can be used
with iterator combinators) and there is a module implementing iterator
combinators in the extended basis library
(http://mlton.org/cgi-bin/viewsvn.cgi/mltonlib/trunk/com/ssh/extended-basis/unstable/public/control/iter.sig?view=auto).

-Vesa Karvonen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap.c
Type: text/x-csrc
Size: 873 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/7483a834/mmap.c
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap.mlb
Type: application/octet-stream
Size: 226 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/7483a834/mmap.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mmap.sml
Type: application/octet-stream
Size: 5053 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/7483a834/mmap-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Build.sh
Type: application/x-sh
Size: 254 bytes
Desc: not available
Url : http://mlton.org/pipermail/mlton-user/attachments/20081011/7483a834/Build.sh