[MLton] Extra GC pressure

[Andreas Rossberg]

Matthew Fluet wrote:
> 
>> The idea being that if you have an SML object that is backed by a C 
>> object (for example SQLite, GTK, etc), you can indicate to the GC how 
>> much memory (approximately) the C-side is consuming. When you free 
>> that memory, you tell the GC that the memory pressure has been 
>> reduced. The goal of this is to help trigger a GC earlier to clean up 
>> SML objects (and thus their attached C state).
>  >
>> A better interface would probably be MLton.GC.alloc <n> which returns 
>> an opaque type. When that type is GC'd, the <n> bytes are additionally 
>> subtracted from the extra memory pressure counter.
> 
> I can imagine the situation you describe, but have you encountered it in 
> practice?

We definitely have encountered the need for this in Alice ML. Our 
binding to the Gecode constraint library uses foreign pointers to fairly 
large C++ objects. For instance, the representation of search spaces can 
consume arbitrary amounts (multi 100M in the extreme) of memory in C 
land, depending on the number of constraint variables and active 
propagators etc. We surely have wished to have a feature along these 
lines for our GC. I can imagine similar problems showing up with other 
non-trivial libraries, e.g. GUIs.

However, the problem is that the alloc-time hint Wesley suggests isn't 
even sufficient, because memory consumption of something like search 
spaces grows over time, by allocating more local objects, e.g. when 
starting propagation and computing a fixpoint. That is, you'd need some 
form of dynamic feedback between C allocation and the ML GC, such that 
memory hints can be adjusted. Unfortunately, we don't have any good idea 
how to cope with it, though, since C libs usually provide no good way to 
enable such feedback. In the case of Gecode, memory management in fact 
is carefully hidden behind abstraction barriers. This is quite similar 
to the "multiple GC" problem.

-- 
Andreas Rossberg, rossberg at ps.uni-sb.de