[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] Compressing large files on Condor
- Date: Fri, 20 Jul 2007 16:31:09 +0300
- From: "Claudiu Udrea" <claudiu@xxxxxxx>
- Subject: Re: [Condor-users] Compressing large files on Condor
Simon has a good point. A great such utility we have been using at out site is 7Zip, mainly because it has really simple command line options. Under Windows you can wrap the job in a batch script and under Linux in a shell script.
The George Washington University
On 7/19/07, Si Hammond
Can you not make one large compressed file say a zip and then use a
script to decompress these when the job is running?
> On Thu, 19 Jul 2007, Alexandra Lupu wrote:
>> My Condor jobs produce huge (64 MB) .fits files in large numbers,
>> and they
>> can take about 4 hours to download to my desktop. Does anyone
>> know of a
>> way to compress these files while they're on the Condor network,
>> so as to
>> reduce the download time?
> My answer is not really related to condor, but it might help...
> 64MB is quite small, we have FITS files of around 10GB.
> imcopy, from the cfitsio distribution, supports RICE compression for
> images, which is quite efficient and fast. But beware that it is
> for floating points (bitpix<0) images.
> Syntax: imcopy toto.fits 'toto.fits.fz[COMPRESS]'
> Condor-users mailing list
> To unsubscribe, send a message to
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> The archives can be found at:
Condor-users mailing list
To unsubscribe, send a message to
firstname.lastname@example.org with a
You can also unsubscribe by visiting
The archives can be found at: