Re: [Condor-users] Compressing large files on Condor

[Si Hammond <simon.hammond@xxxxxxxxx>]

Can you not make one large compressed file say a zip and then use a  
script to decompress these when the job is running?


> On Thu, 19 Jul 2007, Alexandra  Lupu wrote:
> > My Condor jobs produce huge (64 MB) .fits files in large numbers,  
> > and they
> > can take about 4 hours to download to my desktop.  Does anyone  
> > know of a
> > way to compress these files while they're on the Condor network,  
> > so as to
> > reduce the download time?
>
> My answer is not really related to condor, but it might help...
>
> 64MB is quite small, we have FITS files of around 10GB.
> imcopy, from the cfitsio distribution, supports RICE compression for
> images, which is quite efficient and fast. But beware that it is  
> lossless
> for floating points (bitpix<0) images.
>
> Syntax: imcopy toto.fits 'toto.fits.fz[COMPRESS]'
>
> Cheers,
> Fred.
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx  
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/