[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Compressing large files on Condor

Can you not make one large compressed file say a zip and then use a script to decompress these when the job is running?

On Thu, 19 Jul 2007, Alexandra  Lupu wrote:
My Condor jobs produce huge (64 MB) .fits files in large numbers, and they can take about 4 hours to download to my desktop. Does anyone know of a way to compress these files while they're on the Condor network, so as to
reduce the download time?

My answer is not really related to condor, but it might help...

64MB is quite small, we have FITS files of around 10GB.
imcopy, from the cfitsio distribution, supports RICE compression for
images, which is quite efficient and fast. But beware that it is lossless
for floating points (bitpix<0) images.

Syntax: imcopy toto.fits 'toto.fits.fz[COMPRESS]'


Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: