[MLton-user] *** glibc detected *** realloc(): invalid pointer

[Matthew Fluet]

On Tue, 19 Aug 2008, Nicolas Bertolotti wrote:
> I faced the exact same issue with MLton 2005.
>
> This is actually not a bug in GMP but an overflow in the IntInf implementation.
>
> The broken piece of code is located in the file int-inf.c :
> <
> static inline void initRes (__mpz_struct *mpzp, uint bytes) {
>        ...
>        mpzp->_mp_alloc = (gcState.limitPlusSlop - (pointer)bp->limbs) / 4;
>        ...
> }
> <
>
> The overflow occurs on machines with a large amount of RAM (it caused 
> some crashes in our product on some recent 64 bits machines while 
> running the product in 32 bits mode) when the "-" operation overflows 
> and leads to a negative result.

Right.  You need a large heap in order for the difference to come out 
negative.  The _mp_alloc field is declared as 'int'; presumably there is a 
'if (res->_mp_alloc < minSize) { realloc; }' somewhere in the GMP 
implementation.  As a signed comparison, it will always trigger if 
_mp_alloc is negative.

[I still don't quite understand why a 64-bit machine with 32-bit mode 
program is more susceptible than a 32-bit machine with lots of RAM. 
Possibly, a 64-bit kernel frees up more of the 32-bit virtual address 
space, so the program is able to get larger heaps that exhibit the 
overflow.]

> In the latest version of MLton (I think the change is related to the 64 bits port), this piece of code has been replaced with :
> <
> res->_mp_alloc = (s->limitPlusSlop - (pointer)bp->obj.body.limbs) / (sizeof(mp_limb_t));
> <
> Here, the overflow may still occur. Anyway, as the result is then divided by "sizeof(mp_limb_t)" which is an unsigned value, the final result is correct.

The division by an unsigned value causes the dividend to be cast from a 
signed value to an unsigned value.  After the unsigned division, the 
high-bit has to be zero, so the result is a positive number.

> A simple fix for you is to replace < 4 > with < 4U > in your version.

There is another latent overflow present on 64-bit systems: the above 
assignment may overflow because the RHS is 64-bit but the LHS is 32-bit 
(on amd64-linux).  Again, though, one needs a farily large heap to 
trigger the overflow.  SVN r6527 fixed this.

> ________________________________
> From: mlton-user-bounces at mlton.org [mailto:mlton-user-bounces at mlton.org] On Behalf Of Wesley W. Terpstra
> Sent: Tuesday, August 19, 2008 11:48 PM
> To: mlton-user at mlton.org
> Subject: [MLton-user] *** glibc detected *** realloc(): invalid pointer
>
> Has anyone else noticed that gmp seems to be broken as of today? My system hasn't changed, and I downgraded to MLton 20061107 on my completely normal etch machine. However, when I try to execute MLton-generated executables now, I often get the error:
> *** glibc detected *** realloc(): invalid pointer: 0x6e8ecdc8 ***
> which seems to be coming from gmp:
> Program received signal SIGABRT, Aborted.
> 0xffffe405 in __kernel_vsyscall ()
> (gdb) bt
> #0  0xffffe405 in __kernel_vsyscall ()
> #1  0x55609811 in raise () from /lib/tls/i686/cmov/libc.so.6
> #2  0x5560afb9 in abort () from /lib/tls/i686/cmov/libc.so.6
> #3  0x5563edfa in __fsetlocking () from /lib/tls/i686/cmov/libc.so.6
> #4  0x55649063 in realloc () from /lib/tls/i686/cmov/libc.so.6
> #5  0x556490b1 in realloc () from /lib/tls/i686/cmov/libc.so.6
> #6  0x55648dcc in realloc () from /lib/tls/i686/cmov/libc.so.6
> #7  0x55581559 in __gmp_default_reallocate () from /usr/lib/libgmp.so.3
> #8  0x55596916 in __gmpz_realloc () from /usr/lib/libgmp.so.3
> #9  0x55593965 in __gmpz_mul () from /usr/lib/libgmp.so.3
> #10 0x0899c3af in ?? ()
> ...
>
> What I find particularly worrying is that AFAIK, when MLton calls gmpz_mul, gmp should NEVER need to realloc the integer..? Is this some sort of Y2008 bug in gmp? Has anyone else started experiencing it as of today? It doesn't always happen to me, but trying to build MLton seems a sure-fire way to trigger it.
>
>