[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Error in large file transfer : Condor or WinsockBug ?



>On 9/1/06, Christopher Mellen <Chris.Mellen@xxxxxxxxxx> wrote:
>> I'm trying to upload a very large input file, size in excess of 4Gb,
>using
>> Condor 6.6.11 on a cluster of XP machines.

Snip ...

>>
>> Any ideas much appreciated ....
>
>There is a bug in the file transfer on windows in 6.6 series when
>transferring > 2GB files
>in 6.6.11 this was mitigated by allowing the sum total of the files
>transferred to be > 2GB but (IIRC) no individual file can still be
>over 2 GB. (This is all totally fixed on 6.8 series)
>
> am not 100% sure about this though because the bug I found would
>actually *work* on files from 4 to 6 GB since it was an int overflow
>bug.
>
>I suggest you split the input file into 4 files, each 1 GB and try
>transferring that way. If this works you
>* know it's a bug in 6.6.11 and you can move to 6.8 where this should be
>fixed
>* have a work around :) in the meantime
>
>If you are submitting a text file this big have you considered
>compressing it pre transfer and reading it with a automatic
>decompression stream? You may not be able to change your program on
>this front - but if you can may decent libraries exist to do this.
>
>Matt

Thanks for the v. useful info on the 6.6 series bug. Looks like we will have
to roll 6.8 out in the very near future ...

Essentially though we were being careless as the 4Gb files are ASCII text
and so very amenable to compression. Hence our 'workaround' has been to
compress the files before transfer. Of course this is what we should have
been doing all along as file transfer times are now much reduced and job
throughput significantly improved.

Cheers,
Chris.



-- 
Message protected by MailGuard: e-mail anti-virus, anti-spam and content filtering.
http://www.mailguard.com.au/mg