Re: [Condor-users] limit on input file transfers ????

[Matt Hope <matthew.hope@xxxxxxxxx>]

On Thu, 31 Mar 2005 10:28:06 +0100, Dr Ian C. Smith
<i.c.smith@xxxxxxxxxxxxxxx> wrote:
> 
> 
> --On 30 March 2005 12:22 -0600 Zachary Miller <zmiller@xxxxxxxxxxx> wrote:
> 
> > alain's use of rsync, combined with extracting only the useful data from
> > the blast output is quite nice.  i plan on moving our BLAST framework
> > here at the UW to something similar.
> 
> But there's no rysnc for windows AFAIK. 

there exists a cygwin runnable version - I do not recommend it.

Robocopy is excellent if it is capable of accessing the remote files
(SMB share?) it should do the job nicely (/MIR and bob's your uncle)

>There is also the problem of
> permissions. The db files need to be writable for the updates to work
> and thus need to put in the user writable area. This is deliberately
> volatile
> so that junk doesn't accumulate in it. Also having relatively large amounts
> of disk tied up with condor on a semi-permanent basis is unlikely to
> be politically popular.

What you might want to consider is whether or not some users would be
happy to expose that they are willing to do this via startd ads. Then
blast jobs can prefer to run on these machines getting a potentially
big speed up (and reducing your network load to boot)

Plus using robocopy/rsync to send the files is still better than
simply writing repeatedly every single time when running several jobs
even though the data is still there!

> I'll try 6.7

best of luck :) The condor week seemed to feature things like stork
and various other data related niceness - you may find what you are
doing dovetails with some of the current work they are doing and can
benefit from it.

Matt