Re: [Condor-users] limit on input file transfers ????

["Dr Ian C. Smith" <i.c.smith@xxxxxxxxxxxxxxx>]

--On 30 March 2005 09:13 -0600 Nick LeRoy <nleroy@xxxxxxxxxxx> wrote:

> On Wed March 30 2005 7:27 am, Dr Ian C. Smith wrote:
> > Very quick question:
> >
> > Is there a Condor-imposed limit on the total size of
> > input files that can be transfered to an execute host or
> > a limit on the total time it takes to transfer them.
>
> > >         -671628160  -  Run Bytes Received By Job
> >
> > The entire database is ~ 3.5 GB but the largest file in only ~ 500 MB
> > and there is enough free disk space on the execution host. The negative
> > value for the bytes transferred suggests a 32 bit addressing limitation
> > to me.
>
> What version of Condor are you running?  Prior to Condor 6.7.2, Condor
> couldn't handle files larger than 2G, and it's possible that large
> collections of transfers could have caused problems, too (I don't
> remember  all of the details).  These changes never appeared in the 6.6
> series.
>
> -Nick

6.6.5 on all the execution hosts and the submit host, 6.6.9 on the central
manager. Are the input files ZIP'ed (or similar) before staging as with 
Globus -
I can imagine this could cause problems with a 2GB limit. Any way around 
this ?

We are looking at making the databases available through remote file system
access ( Novell or similar ). I suspect that the overhead involved in that
may be more than for actually copying the files over though - particularly 
if the
file staging time can be amortized over many BLAST runs. I think it should 
be possible
to transfer individual divisions to separate execution hosts on a 
divide-and-rule
basis. More programming hassle though :-(


cheers,

-ian.