Re: [Condor-users] Compressing large files on Condor

["Claudiu Udrea" <claudiu@xxxxxxx>]

Simon has a good point. A great such utility we have been using at out site is 7Zip, mainly because it has really simple command line options. Under Windows you can wrap the job in a batch script and under Linux in a shell script.

 

Claudiu Udrea

The George Washington University

Washington DC

On 7/19/07, Si Hammond <simon.hammond@xxxxxxxxx> wrote:

Can you not make one large compressed file say a zip and then use a
script to decompress these when the job is running?


>
> On Thu, 19 Jul 2007, Alexandra  Lupu wrote:
>> My Condor jobs produce huge (64 MB) .fits files in large numbers,
>> and they
>> can take about 4 hours to download to my desktop.  Does anyone
>> know of a
>> way to compress these files while they're on the Condor network,
>> so as to
>> reduce the download time?
>
> My answer is not really related to condor, but it might help...
>
> 64MB is quite small, we have FITS files of around 10GB.
> imcopy, from the cfitsio distribution, supports RICE compression for
> images, which is quite efficient and fast. But beware that it is
> lossless
> for floating points (bitpix<0) images.
>
> Syntax: imcopy toto.fits 'toto.fits.fz[COMPRESS]'
>
> Cheers,
> Fred.
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxx .edu
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu /mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/

_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu /mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/