After so many tries, I had thought I was done with it. Well the virtual memory limit on my machine still stands so when the shellSort encountered an array of 1.5 million records, it failed. I could have gone ahead with getting the VM size increased but decided it was better to deal with the limitation as it stood. Who knows, the next bloke may not have much virtual memory either (though, I suppose, he's more likely to have lots more.)
I handed the job over to SORT.EXE and used the Exec method to call it. Like this:
The actual call to SortFile is what were talking about but the oSamples stuff needs some explaining.
SuperContainer is a class based on the ADODB.Streams object.
Interestingly I had a lot of trouble with the ReadFromFile method in ADODB.Stream. I think this had to do with it expecting Unicode but getting ASCII. When you hand a Unicode file over to SORT.EXE, it sorts it okay but changes it into ASCII. ReadFromFile tries to interpret the file as Unicode and you end up with a mess. Thus the GetFile()/Split/For Each thing in my ReadFile method.
The code for the sorting routine is:
I have a global in my stdFunctions.vbs called bDebugging. It's normally False but I can change it and get some useful feedback thereby.
The upshot of all this is that the sorting seems to be working ... but still not perfectly -- when I gave SORT.EXE a 63 Gigabyte file it didn't like it. But the likelihood of generating datasets that large is low.