From Nicolas.Bertolotti at mathworks.fr Tue Sep 1 09:16:19 2009 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Tue Sep 1 09:17:13 2009 Subject: [MLton] RE: Crash during cheney-copy on Windows In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA3DB4AC3F1C@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA3DB4AC3F1C@EXCHANGE-UK.ad.mathworks.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA7AAE995FE8@EXCHANGE-UK.ad.mathworks.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2747 bytes Desc: image001.jpg Url : http://mlton.org/pipermail/mlton/attachments/20090901/4b0cc624/image001-0001.jpg From Nicolas.Bertolotti at mathworks.fr Tue Sep 1 10:17:00 2009 From: Nicolas.Bertolotti at mathworks.fr (Nicolas Bertolotti) Date: Tue Sep 1 10:17:37 2009 Subject: [MLton] Crash during cheney-copy on Windows In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA3DB4AC410B@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA3DB4AC3F1C@EXCHANGE-UK.ad.mathworks.com> <49944C50.5010401@cmu.edu> <8320D98DA9A5C54C926D397795FE7CEA3DB4AC410B@EXCHANGE-UK.ad.mathworks.com> Message-ID: <8320D98DA9A5C54C926D397795FE7CEA7AAE996004@EXCHANGE-UK.ad.mathworks.com> I am facing a sporadic signal 11 (segmentation fault) on Linux which could be caused by the same bug. After enabling assertions, I could identify that there was a sporadic assertion failure in updateCrossMap() : void updateCrossMap (GC_state s) { GC_cardMapIndex cardIndex; pointer cardStart, cardEnd; ? cardEnd = cardStart + CARD_SIZE; loopObjects: assert (objectStart < oldGenEnd); <= this assertion may sporadically fail assert ((objectStart == s->heap.start or cardStart < objectStart) and objectStart <= cardEnd); ? The assertion fails during the execution of a minor cheney copy that occurs after 2 calls to GC_pack() and the heap size did not change during the execution of the 2nd GC_pack(). I finally identified that the SVN revision r6776 introduced a change that was motivated by the fact we need to clear the cross map after every major GC. But, if we look at the code, we can see that the cross map is only cleared when the ?mayResize? flag is set (and, as a matter of fact, this flag is not set by GC_pack()) : void majorGC (GC_state s, size_t bytesRequested, bool mayResize) { ? if (mayResize) { resizeHeap (s, s->lastMajorStatistics.bytesLive + bytesRequested); setCardMapAndCrossMap (s); } ? } As the revision r6776 also introduces the removal of some calls to clearCrossMap() which were performed systematically at the end of a major cheney-copy or major mark-compact, it seems to me that the call to setCardMapAndCrossMap(s) should actually always be performed (or maybe adding a else { clearCrossMap(s); } would be enough). I moved the call to setCardMapAndCrossMap(s) after the if and it seems to solve the issue (anyway, as it was sporadic, I am not so sure) What do you think ? Nicolas > -----Original Message----- > From: mlton-bounces@mlton.org [mailto:mlton-bounces@mlton.org] On > Behalf Of Nicolas Bertolotti > Sent: Thursday, February 12, 2009 7:56 PM > To: Daniel Spoonhower > Cc: mlton@mlton.org > Subject: RE: [MLton] Crash during cheney-copy on Windows > > > It's not clear to me exactly what debugging you are able to enable > and > > still observe the problem. I believe the most important check would > > be "invariantForGC" which is run at the beginning and end of each > > collection. Are you able to run this function and still observe the > > problem? (It is in the debug version of the runtime and is also > > enabled by -DASSERT=1.) > > The assertions are enabled so invariantForGC() is called and does not > reveal anything. > > > > > If you are suspicious of old data in the heap, you could try > > explicitly clearing the new heap at the beginning of a Cheney copy > > (i.e. in majorCheneyCopyGC). > > I examined the array allocation routine and it definitely properly > resets the contents of all the cells. > > It is also not a limit case (such as a '<' instead of a '<=' somewhere) > either as the crash appears to occur around the cell 15000 of a 32768 > cells array. > > Still investigating ... > > > > > > > --djs > > > > Nicolas Bertolotti wrote: > > > Hello, > > > > > > > > > > > > I am currently facing a crash of my product during the cheney-copy > > > operation on Windows. This crash is very hard to reproduce as it is > > very > > > volatile (some slight changes in the SML code make it disappear ; > it > > > depends on the memory amount etc?). > > > > > > > > > > > > I finally could activate some debug messages and assertions (it is > > not > > > full the debugging mode because enabling it causes the issue to > > disappear) : > > > > > > [GC: Starting gc #73; requesting 512 nursery bytes and 0 old-gen > > bytes,] > > > > > > [GC: heap at 0x31880000 of size 710,967,296 bytes,] > > > > > > [GC: with nursery of size 617,405,820 bytes (86.8% of heap),] > > > > > > [GC: and old-gen of size 93,561,476 bytes (13.2% of heap),] > > > > > > ? > > > > > > [GC: Starting major Cheney-copy;] > > > > > > [GC: from heap at 0x31880000 of size 710,967,296 bytes,] > > > > > > [GC: to heap at 0x08fc0000 of size 710,967,296 bytes.] > > > > > > ? > > > > > > [GC: Finished major Cheney-copy; copied 97,279,788 bytes.] > > > > > > ? > > > > > > [GC: Starting gc #77; requesting 512 nursery bytes and 0 old-gen > > bytes,] > > > > > > [GC: heap at 0x08fc0000 of size 710,967,296 bytes,] > > > > > > [GC: with nursery of size 612,833,480 bytes (86.2% of heap),] > > > > > > [GC: and old-gen of size 98,133,816 bytes (13.8% of heap),] > > > > > > ? > > > > > > [GC: Starting major Cheney-copy;] > > > > > > [GC: from heap at 0x08fc0000 of size 710,967,296 bytes,] > > > > > > [GC: to heap at 0x31880000 of size 710,967,296 bytes.] > > > > > > ? > > > > > > foreachObjptrInObject (0x318c2318) header = 000004c7 tag = ARRAY > > > bytesNonObjptrs = 0 numObjptrs = 1 > > > > > > forwardObjptr opp = 0x318c2318 op = 0x00000000091b7714 p = > > 0x091b7714 > > > > > > forwardObjptr --> *opp = 0x319f57dc > > > > > > ? > > > > > > forwardObjptr opp = 0x318d1354 op = 0x00000000091d78c4 p = > > 0x091d78c4 > > > > > > forwardObjptr --> *opp = 0x31a173ac > > > > > > forwardObjptr opp = 0x318d1358 op = 0x00000000318da094 p = > > 0x318da094 > > > > > > Assertion failed at line 58 of file gc/object.c > > > > > > ===> corresponds to the ?assert (1 == (header & > > GC_VALID_HEADER_MASK));? > > > at the beginning of the splitHeader() function > > > > > > > > > > > > Using those debug messages, we can see that all calls to > > forwardObjptr() > > > are performed on objects whose address is, as expected, in the > ?from? > > > heap whereas the last call that leads to a crash receives an > invalid > > > pointer whose address is in the ?to? heap. > > > > > > > > > > > > We also see that the 2 heaps have been allocated during a previous > > call > > > to the garbage collector and a previous ?cheney-copy? has already > > been > > > performed between those. > > > > > > > > > > > > I suspect that maybe a previous GC operation left some old pointers > > in > > > one of the heaps and those have not been properly cleared during an > > > object allocation or so on. > > > > > > > > > > > > In any case, I simply don?t know what to do in order to identify > the > > > root cause of the issue. Any hint ? > > > > > > > > > > > > Best regards > > > > > > > > > > > > cid:image001.gif@01C7BFD3.87CF8F80 > > > > > > > > > > > > > > > > > > Accelerating the pace of engineering and science > > > > > > > > *Nicolas Bertolotti* > > > Senior Development Engineer > > > > > > > > > > > > 2 Rue de Paris > > > 92196 Meudon Cedex > > > > > > France > > > > > > Nicolas.Bertolotti@mathworks.fr > > > > > > > > > > > > > > tel: > > > fax: > > > mobile: > > > > > > > > > > > > +33.1.41.14.88.55 > > > > > > +33.1.55.64.06.64 > > > > > > +33.6.86.41.87.15 > > > > > > > > > > > > > > > ------------------------------------------------------------------- > - > > > - > > --- > > > > > > _______________________________________________ > > > MLton mailing list > > > MLton@mlton.org > > > http://mlton.org/mailman/listinfo/mlton From wesley at terpstra.ca Wed Sep 2 07:09:12 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Sep 2 07:09:49 2009 Subject: [MLton] MLtonIntInf.rep Message-ID: <162de7480909020709s58bf11d5ta9029459913146b2@mail.gmail.com> There is already this in MLtonIntInf: *structure* BigWord : WORD *structure* SmallInt : INTEGER *datatype** rep *=* Big of BigWord.word vector *|* Small of SmallInt.int **val* rep: t -> rep This is pretty useful since it makes it possible to convert an IntInf into a byte-stream or hexadecimal in linear-time. Without this API the fastest algorithm I know requires nlog(n) time using >> recursively. However, I also need to be able to convert the other direction too; from rep to t. I can see this is easy enough to do with Prim.fromVector. Does anyone have an objection to the addition of 'fromRep' to MLtonIntInf? Otherwise loading an IntInf from the network will have to be slow. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090902/b672c566/attachment.html From wesley at terpstra.ca Wed Sep 2 07:46:22 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Sep 2 07:46:28 2009 Subject: [MLton] Re: MLtonIntInf.rep In-Reply-To: <162de7480909020709s58bf11d5ta9029459913146b2@mail.gmail.com> References: <162de7480909020709s58bf11d5ta9029459913146b2@mail.gmail.com> Message-ID: <162de7480909020746w1b9ee3bbmcdd96418212cb0c2@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: fromRep.patch Type: text/x-patch Size: 1856 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090902/a8d8b4f8/fromRep.bin From wesley at terpstra.ca Wed Sep 9 13:41:24 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Wed Sep 9 13:42:04 2009 Subject: [MLton] allocateRegisters.toLiveness absurdly slow Message-ID: <162de7480909091341p48e1617q7343f2faf35a4ba9@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: log Type: application/octet-stream Size: 51332 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090909/15f04c56/log-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: profile Type: application/octet-stream Size: 24117 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090909/15f04c56/profile-0001.obj From mtf at cs.rit.edu Fri Sep 11 07:44:58 2009 From: mtf at cs.rit.edu (Matthew Fluet) Date: Fri Sep 11 07:45:04 2009 Subject: [MLton] Re: [MLton-commit] r7223 In-Reply-To: References: Message-ID: On Sat, 5 Sep 2009, Wesley Terpstra wrote: > Where there is a "to", there should be a "from". > > Converting an intinf to/from a byte-stream is a pretty important operation > in cryptography. The basis library doesn't provide a way to convert to bytes > aside from >> and WordX.fromLargeInt. It is not possible to convert an integer > to bytes in linear time with these operations. Therefore, this patch extends > the MLton extension to converts IntInf to/from it's underlying representation. Looks good, especially with the checking for valid representations. Don't forget to update http://mlton.org/MLtonIntInf with signature changes. From wesley at terpstra.ca Sun Sep 13 12:49:21 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Sun Sep 13 12:49:26 2009 Subject: [MLton] Re: allocateRegisters.toLiveness absurdly slow In-Reply-To: <162de7480909091341p48e1617q7343f2faf35a4ba9@mail.gmail.com> References: <162de7480909091341p48e1617q7343f2faf35a4ba9@mail.gmail.com> Message-ID: <162de7480909131249x6bc8c6feif0bac159c0c57533@mail.gmail.com> On Wed, Sep 9, 2009 at 10:41 PM, Wesley W. Terpstra wrote: > I have a project where allocateRegisters.toLiveness takes 220s out of a > total build time of 240s. I'm not sure what exactly causes this... > In case it's not obvious, this only happens with the x86/amd64 codegens. C and bytecode compile just fine. I'd try and solve it myself, but I haven't a clue where to even begin. That 240s total was on the fastest machine available to me; on my laptop it takes 20 minutes... What algorithm is toLiveness using? Does it have exponential complexity in the number of variables? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090913/dcfa5570/attachment.html From wesley at terpstra.ca Mon Sep 14 04:49:26 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Sep 14 04:50:00 2009 Subject: [MLton] Re: allocateRegisters.toLiveness absurdly slow In-Reply-To: <162de7480909131249x6bc8c6feif0bac159c0c57533@mail.gmail.com> References: <162de7480909091341p48e1617q7343f2faf35a4ba9@mail.gmail.com> <162de7480909131249x6bc8c6feif0bac159c0c57533@mail.gmail.com> Message-ID: <162de7480909140449r5ab8b3a6me4eba9640aef740a@mail.gmail.com> On Sun, Sep 13, 2009 at 9:49 PM, Wesley W. Terpstra wrote: > Does it have exponential complexity in the number of variables? > The answer to this question appears to be: YES! I've found a point in my project where adding one element to the record (more than) doubles the compile time. I'm using the FunctionalRecordUpdate pattern with this record. At this point my compile times have gone up to 25 minutes on the fastest computer available to me (my laptop now takes 1.4 hours)... It had 22 elements for a compile time of 10:30 minutes then I increased it to 23 elements (plus some code to force the element to be used) and the compile time went up to 24:46 minutes At least now that I know what causes the problem, I can work around it, but it seems pretty unfortunate that MLton has an exponential time liveness calculation. Can this be fixed? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090914/d1819a84/attachment.htm From wesley at terpstra.ca Mon Sep 14 08:36:22 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Sep 14 08:37:01 2009 Subject: [MLton] Re: allocateRegisters.toLiveness absurdly slow In-Reply-To: <162de7480909140449r5ab8b3a6me4eba9640aef740a@mail.gmail.com> References: <162de7480909091341p48e1617q7343f2faf35a4ba9@mail.gmail.com> <162de7480909131249x6bc8c6feif0bac159c0c57533@mail.gmail.com> <162de7480909140449r5ab8b3a6me4eba9640aef740a@mail.gmail.com> Message-ID: <162de7480909140836n7f9ec7fhe116a93dee66a789@mail.gmail.com> On Mon, Sep 14, 2009 at 1:49 PM, Wesley W. Terpstra wrote: > On Sun, Sep 13, 2009 at 9:49 PM, Wesley W. Terpstra wrote: > >> Does it have exponential complexity in the number of variables? >> > > The answer to this question appears to be: YES! > > I've found a point in my project where adding one element to the record > (more than) doubles the compile time. I'm using the FunctionalRecordUpdate > pattern with this record. At this point my compile times have gone up to 25 > minutes on the fastest computer available to me (my laptop now takes 1.4 > hours)... It had 22 elements for a compile time of 10:30 minutes then I > increased it to 23 elements (plus some code to force the element to be used) > and the compile time went up to 24:46 minutes > > At least now that I know what causes the problem, I can work around it, but > it seems pretty unfortunate that MLton has an exponential time liveness > calculation. Can this be fixed? > Good lord. Don't blame MLton, blame sweeks and vesa! The FunctionalRecordUpdate on the wiki has exponential type explosion. This makes it pretty useless in practice. Also, the code size still grows quadratically due to the list of 'A's that one has to make. I've made a work-alike version that has linear code growth and no explosion. This is it: (* Briefly, if you have a record { foo: int, bar: real, baz: word }... * fun updateRecord z = * let * fun from v1 v2 v3 = {foo = v1, bar = v2, baz = v3} * fun to f {foo = v1, bar = v2, baz = v3} = f v1 v2 v3 * in * makeUpdate3 (from, from, to) z * end * * You can then update records with: updateRecord record (set#bar 1.0) $ *) structure FunctionalRecordUpdate = struct local (* The user supplies us with these two functions: * to f { a, b, c } = f a b c * from a b c = { a, b, c } * * The goal is to create a family of functions like: * f0 (f, z) a b c = f (z a) b c * f1 (f, z) a b c = f a (z b) c * f2 (f, z) a b c = f a b (z c) * * These functions are passed as arguments via selector: * c3 g = g f0 f1 f2 * * We can thus set g=from to get a record of functional updates. * Use the update function as: to (record, upd (from, z)) *) fun next g (f, z) x = g (f x, z) fun f1 (f, z) x = f (z x) fun f2 z = next f1 z fun f3 z = next f2 z fun f4 z = next f3 z fun f5 z = next f4 z fun f6 z = next f5 z fun f7 z = next f6 z -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090914/bde82dc6/attachment.html From wesley at terpstra.ca Mon Sep 14 08:37:49 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Sep 14 08:37:54 2009 Subject: [MLton] Re: allocateRegisters.toLiveness absurdly slow In-Reply-To: <162de7480909140836n7f9ec7fhe116a93dee66a789@mail.gmail.com> References: <162de7480909091341p48e1617q7343f2faf35a4ba9@mail.gmail.com> <162de7480909131249x6bc8c6feif0bac159c0c57533@mail.gmail.com> <162de7480909140449r5ab8b3a6me4eba9640aef740a@mail.gmail.com> <162de7480909140836n7f9ec7fhe116a93dee66a789@mail.gmail.com> Message-ID: <162de7480909140837q731dd9dcx4fd7383363e362a8@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: functional-record-update.sml Type: application/smil Size: 5178 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090914/00683b45/functional-record-update.smil From wesley at terpstra.ca Mon Sep 14 12:07:26 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Sep 14 12:08:00 2009 Subject: [MLton] Bug: Word64.ror with offset 0w32 fails on x86 Message-ID: <162de7480909141207tc52c2d6p65f023d72b7d7537@mail.gmail.com> val x = valOf (Word64.fromString "1234567890abcdef") val y = MLton.Word64.ror (x, 0w32) val z = Word64.orb (Word64.<< (x, 0w32), Word64.>> (x, 0w32)) val () = print (Word64.toString y ^ "\n") val () = print (Word64.toString z ^ "\n") Output: 1234567890ABCDEF 90ABCDEF12345678 Expected output: 90ABCDEF12345678 90ABCDEF12345678 I've checked that this bug is NOT from gcc by compiling the following: #include #include uint64_t Word64_ror(uint64_t w1, uint32_t w2); int main () { uint64_t x = Word64_ror(0x1234567890ABCDEFUL, 32); printf ("%llX\n", x); return 0; } gcc -Wall -O2 bug.c /usr/lib/mlton/self/libmlton.a -o bug Output: 90ABCDEF12345678 The number 0w32 strikes me as a case that someone probably optimized incorrectly. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090914/50a1e3f8/attachment.htm From wesley at terpstra.ca Mon Sep 14 12:48:48 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Sep 14 12:49:24 2009 Subject: [MLton] Re: Bug: Word64.ror with offset 0w32 fails on x86 In-Reply-To: <162de7480909141207tc52c2d6p65f023d72b7d7537@mail.gmail.com> References: <162de7480909141207tc52c2d6p65f023d72b7d7537@mail.gmail.com> Message-ID: <162de7480909141248i452ed774p598184061e12dd78@mail.gmail.com> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: rolr-bug.patch Type: text/x-patch Size: 2126 bytes Desc: not available Url : http://mlton.org/pipermail/mlton/attachments/20090914/398a2cec/rolr-bug.bin From wesley at terpstra.ca Mon Sep 14 13:25:18 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Mon Sep 14 13:25:57 2009 Subject: [MLton] Re: allocateRegisters.toLiveness absurdly slow In-Reply-To: <162de7480909140836n7f9ec7fhe116a93dee66a789@mail.gmail.com> References: <162de7480909091341p48e1617q7343f2faf35a4ba9@mail.gmail.com> <162de7480909131249x6bc8c6feif0bac159c0c57533@mail.gmail.com> <162de7480909140449r5ab8b3a6me4eba9640aef740a@mail.gmail.com> <162de7480909140836n7f9ec7fhe116a93dee66a789@mail.gmail.com> Message-ID: <162de7480909141325i3b943f49le6129f07cc5d9afe@mail.gmail.com> On Mon, Sep 14, 2009 at 5:36 PM, Wesley W. Terpstra wrote: > Good lord. Don't blame MLton, blame sweeks and vesa! > I've updated the wiki at . Hopefully other people can avoid the problems I had. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090914/aa3eb40d/attachment.html From mtf at cs.rit.edu Tue Sep 15 07:11:34 2009 From: mtf at cs.rit.edu (Matthew Fluet) Date: Tue Sep 15 07:11:41 2009 Subject: [MLton] Re: [MLton-commit] r7224 In-Reply-To: References: Message-ID: On Mon, 14 Sep 2009, Wesley Terpstra wrote: > The applyPrim function incorrectly optimized rol/ror. > > The problem stems from confusing the wordsize of the left and right arguments. > > The optimization should check if the rol/ror is a multiple of the left > wordsize, replacing the rol/ror by identity. However, as it incorrectly took > the right wordsize, a shift by 32 was considered identity for a 64 bit value. Looks like the bug was introduced with r2125 (20030623), which added multiple Int, Real, and Word sizes to the compiler. The shift amount has always been a Word32, but in the absence of 64bit values, this simply meant that we weren't optimizing rol/ror on Word16 and Word8 for mod16 and mod8 values respectively. From mtf at cs.rit.edu Wed Sep 16 06:14:35 2009 From: mtf at cs.rit.edu (Matthew Fluet) Date: Wed Sep 16 06:14:40 2009 Subject: [MLton] RE: Crash during cheney-copy on Windows In-Reply-To: <8320D98DA9A5C54C926D397795FE7CEA7AAE995FE8@EXCHANGE-UK.ad.mathworks.com> References: <8320D98DA9A5C54C926D397795FE7CEA3DB4AC3F1C@EXCHANGE-UK.ad.mathworks.com> <8320D98DA9A5C54C926D397795FE7CEA7AAE995FE8@EXCHANGE-UK.ad.mathworks.com> Message-ID: On Tue, 1 Sep 2009, Nicolas Bertolotti wrote: > I am facing a sporadic signal 11 (segmentation fault) on Linux which > could be caused by the same bug. I'm not sure this bug would cause the previous failure. An invalid card/cross map would cause problems for a minor gc, but a major gc ignores the card/cross map. The previous failure occured during a major gc. Of course, an invalid card/cross map might cause the minor gc to leave the heap in a bad state that triggers the failure in the major gc, but I would expect the failure to come during the minor gc. > I finally identified that the SVN revision r6776 introduced a change > that was motivated by the fact we need to clear the cross map after > every major GC. But, if we look at the code, we can see that the cross > map is only cleared when the 'mayResize' flag is set (and, as a matter > of fact, this flag is not set by GC_pack()) : > void majorGC (GC_state s, size_t bytesRequested, bool mayResize) { > ... > if (mayResize) { > resizeHeap (s, s->lastMajorStatistics.bytesLive + bytesRequested); > setCardMapAndCrossMap (s); > } > ... > } > As the revision r6776 also introduces the removal of some calls to > clearCrossMap() which were performed systematically at the end of a > major cheney-copy or major mark-compact, it seems to me that the call to > setCardMapAndCrossMap(s) should actually always be performed (or maybe > adding a else { clearCrossMap(s); } would be enough). Your analysis looks completely right. I will note that a major Cheney Copy GC does clear the card/cross map (via swapHeapsForCheneyCopy), so it is only a GC_pack that induces a Mark Compact GC that would leave the card/cross maps in an invalid state. Performing GC_packs in close succession is likely to induce a Mark Compact GC (since the packed heap isn't big enough for a Cheney Copy). Further investigation might find a way to avoid unnecessarily setting and clearing the maps, but your solution to unconditionally setCardMapAndCrossMap at the end of a majorGC is certainly correct and expedient, so I've committed it (SVN r7228). It simply performs setCardMapAndCrossMap twice in the case of a major Cheney Copy; it might be possible to eliminate the setCardMapAndCrossMap call from swapHeapsForCheneyCopy, if there are no assertions that the current heap and the current card/cross maps agree between the swapHeapsForCheneyCopy and the unconditional setCardMapAndCrossMap at the end of majorGC; there don't appear to be any. SVN r6776 was attempting to systematically setCardMapAndCrossMap immediately after any change the heap pointer or the heap size, but that can probably be relaxed with respect to the swapHeapsForCheneyCopy. From jhr at cs.uchicago.edu Thu Sep 17 07:40:57 2009 From: jhr at cs.uchicago.edu (John Reppy) Date: Thu Sep 17 07:41:32 2009 Subject: [MLton] building mlton on Snow Leopard Message-ID: I'm trying to compile mlton (revision 7228) using the command make TARGET_ARCH=x86_64 smlnj-mlton on Snow Leopard, but I'm hitting type errors in ssa/combine- conversions.fun ssa/combine-conversions.fun:57.5-58.60 Warning: type vars not generalized because of value restriction are instantiated to dummy types (X1,X2,...) ssa/combine-conversions.fun:64.21-64.46 Error: operator and operand don't agree [tycon mismatch] operator domain: ?.S.t * ?.X1 option operand: ?.S.t * (('Z * 'Y * {signed:'X}) * 'W) option in expression: set (x3,SOME conversion) ssa/combine-conversions.fun:68.7-76.20 Error: case object and rules don't agree [tycon mismatch] rule domain: ((?.S.t * 'Z * {signed:bool}) * 'Y) option object: ?.X1 option in expression: (case (get x2) of NONE => stop () | SOME (prev as (,)) => if ( <= ) andalso ( <= ) then loop prev else if andalso then loop else if then else ) ssa/combine-conversions.fun:93.10-100.21 Error: case object and rules don't agree [tycon mismatch] rule domain: ((?.S.t * ?.S.t * {signed:bool}) * ?.S.t) option option object: ?.X1 option Option.t in expression: (case (Option.map (var,get)) of SOME (SOME (,)) => if WordSize.equals (,) then Var x2 else PrimApp {args=Vector.new1 ,prim=Prim.wordExtdToWord , targs=Vector.new0 } | _ => exp) From mtf at cs.rit.edu Thu Sep 17 08:16:23 2009 From: mtf at cs.rit.edu (Matthew Fluet) Date: Thu Sep 17 08:16:29 2009 Subject: [MLton] building mlton on Snow Leopard In-Reply-To: References: Message-ID: On Thu, 17 Sep 2009, John Reppy wrote: > I'm trying to compile mlton (revision 7228) using the command > > make TARGET_ARCH=x86_64 smlnj-mlton > > on Snow Leopard, but I'm hitting type errors in ssa/combine-conversions.fun Nothing to do with Snow Leopard. Everything to do with SML/NJ using an overly restrictive context for type inference. > ssa/combine-conversions.fun:57.5-58.60 Warning: type vars not generalized > because of value restriction are instantiated to dummy types (X1,X2,...) The code in question is: val { get, set, ... } = Property.getSetOnce (Var.plist, Property.initConst NONE) Yes, the type of NONE (and get and set) are undetermined by this declaration. However, subsequent use of get and set within the module determine the types. For a smaller example: structure S = struct val z = (fn x => x) [] val y = z :: [true] :: nil end Accepted by MLton, Poly/ML, and HaMLet; only rejected by SML/NJ. Another example: structure S : sig val z : bool list end = struct val z = (fn x => x) [] end Accepted by MLton, Poly/ML, and HaMLet; only rejected by SML/NJ. Admittedly, we have attempted to keep the MLton sources compatible with SML/NJ, so we will probably put in an explicit type constraint (you'll see such a constraint in most uses of Property.* functions), but it's really SML/NJ's bug. From jhr at cs.uchicago.edu Thu Sep 17 08:28:15 2009 From: jhr at cs.uchicago.edu (John Reppy) Date: Thu Sep 17 08:28:48 2009 Subject: [MLton] building mlton on Snow Leopard In-Reply-To: References: Message-ID: <0A48AD2D-D825-4FA1-B30A-6FB4B162F60B@cs.uchicago.edu> I understand the source of the error, but what is the correct type here so that I can fix it? A debate about the interpretation of the SML'97 semantics is not useful (even though I actually agree with your position). - John On Sep 17, 2009, at 10:16 AM, Matthew Fluet wrote: > On Thu, 17 Sep 2009, John Reppy wrote: >> I'm trying to compile mlton (revision 7228) using the command >> >> make TARGET_ARCH=x86_64 smlnj-mlton >> >> on Snow Leopard, but I'm hitting type errors in ssa/combine- >> conversions.fun > > Nothing to do with Snow Leopard. Everything to do with SML/NJ using > an overly restrictive context for type inference. > >> ssa/combine-conversions.fun:57.5-58.60 Warning: type vars not >> generalized because of value restriction are instantiated to dummy >> types (X1,X2,...) > > The code in question is: > > val { get, set, ... } = > Property.getSetOnce (Var.plist, Property.initConst NONE) > > Yes, the type of NONE (and get and set) are undetermined by this > declaration. However, subsequent use of get and set within the > module determine the types. > > For a smaller example: > > structure S = > struct > val z = (fn x => x) [] > val y = z :: [true] :: nil > end > > Accepted by MLton, Poly/ML, and HaMLet; only rejected by SML/NJ. > > Another example: > > structure S : sig val z : bool list end = > struct > val z = (fn x => x) [] > end > > Accepted by MLton, Poly/ML, and HaMLet; only rejected by SML/NJ. > > > Admittedly, we have attempted to keep the MLton sources compatible > with SML/NJ, so we will probably put in an explicit type constraint > (you'll see such a constraint in most uses of Property.* functions), > but it's really SML/NJ's bug. > From mtf at cs.rit.edu Thu Sep 17 08:35:34 2009 From: mtf at cs.rit.edu (Matthew Fluet) Date: Thu Sep 17 08:35:49 2009 Subject: [MLton] building mlton on Snow Leopard In-Reply-To: <0A48AD2D-D825-4FA1-B30A-6FB4B162F60B@cs.uchicago.edu> References: <0A48AD2D-D825-4FA1-B30A-6FB4B162F60B@cs.uchicago.edu> Message-ID: val { get : Var.t -> ((WordSize.t * WordSize.t * {signed:bool}) * Var.t) option, set, ... } = Property.getSetOnce (Var.plist, Property.initConst NONE) Committed as r7229. On Thu, 17 Sep 2009, John Reppy wrote: > I understand the source of the error, but what is the correct type here so > that I can > fix it? A debate about the interpretation of the SML'97 semantics is not > useful (even > though I actually agree with your position). > > - John > > On Sep 17, 2009, at 10:16 AM, Matthew Fluet wrote: > >> On Thu, 17 Sep 2009, John Reppy wrote: >>> I'm trying to compile mlton (revision 7228) using the command >>> >>> make TARGET_ARCH=x86_64 smlnj-mlton >>> >>> on Snow Leopard, but I'm hitting type errors in >>> ssa/combine-conversions.fun >> >> Nothing to do with Snow Leopard. Everything to do with SML/NJ using an >> overly restrictive context for type inference. >> >>> ssa/combine-conversions.fun:57.5-58.60 Warning: type vars not generalized >>> because of value restriction are instantiated to dummy types (X1,X2,...) >> >> The code in question is: >> >> val { get, set, ... } = >> Property.getSetOnce (Var.plist, Property.initConst NONE) >> >> Yes, the type of NONE (and get and set) are undetermined by this >> declaration. However, subsequent use of get and set within the module >> determine the types. >> >> For a smaller example: >> >> structure S = >> struct >> val z = (fn x => x) [] >> val y = z :: [true] :: nil >> end >> >> Accepted by MLton, Poly/ML, and HaMLet; only rejected by SML/NJ. >> >> Another example: >> >> structure S : sig val z : bool list end = >> struct >> val z = (fn x => x) [] >> end >> >> Accepted by MLton, Poly/ML, and HaMLet; only rejected by SML/NJ. >> >> >> Admittedly, we have attempted to keep the MLton sources compatible with >> SML/NJ, so we will probably put in an explicit type constraint (you'll see >> such a constraint in most uses of Property.* functions), but it's really >> SML/NJ's bug. >> > > > _______________________________________________ > MLton mailing list > MLton@mlton.org > http://mlton.org/mailman/listinfo/mlton -- ============================================ Matthew Fluet Assistant Professor Department of Computer Science Rochester Institute of Technology 102 Lomb Memorial Drive Rochester, NY 14623-5608 Tel: +1.585.475.2854 ; Fax: +1.585.475.4935 mtf@cs.rit.edu ; http://www.cs.rit.edu/~mtf ============================================ CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender (mtf@cs.rit.edu) and destroy any copies of this information. From wesley at terpstra.ca Fri Sep 25 13:26:54 2009 From: wesley at terpstra.ca (Wesley W. Terpstra) Date: Fri Sep 25 13:27:28 2009 Subject: [MLton] Optimization pass: eliminate-replace Message-ID: <162de7480909251326i53f09027n9f9989a7bfbcff10@mail.gmail.com> I am considering an optimization pass to run after refFlatten that turns: val y = !x val () = x := y into a noop. I think the best way to do this is run a DFS and tag refs with a serial number. A deref (!) records the serial number on the assigned variable and any reassignments of that variable. Finally, if a reference is written to (:=), check the variable to see if it is a load of this variable with identical serial number, if so, remove the assignment. If not, increase the serial number. That way in val y = !x val () = x := !x + 1 val z = !x val a = !x val () = x := z val () = x := a val () = x := y the 2nd and 3rd assignments get eliminated, but not 1st and 4th. This optimization is specifically a follow-up optimization to DeepFlatten. If DeepFlatten turned a ref tuple into a record, chances are that a lot of the fields are unmodified when it gets rewritten. In my particular application only 1-3 fields out of 21 or more get changed, so this would be a good speed-up for me at least. Any comments? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mlton.org/pipermail/mlton/attachments/20090925/45f9a36b/attachment.html