Re: [Condor-users] limit on input file transfers ????

[Nick LeRoy <nleroy@xxxxxxxxxxx>]

On Wed March 30 2005 10:10 am, Alain EMPAIN wrote:
> Hello Ian,
>
> I had the same problem, sending blast requests on the whole human genome.
>
> 1/ I have access to all the computing nodes (kept within our intranet)
>    (no comparison with remote Globus machines)
>
> 2/ I installed on every condor execute node a /home/gridcache directory
> and every blast job starts with a 'rsync' command, to be sure that the
> current blast db kept on NFS is well synchronized with its cached local
> counterpart. When no change is observed, it costs only some ms, with
> roughly no network traffic.
>
> 3/ the verbous outputs of blast are kept localy on /tmp, ans only the
> pertinent data are extracted to a table, gzipped and sent back to the
> condor shared file system (NFS).
>
> Perhaps some ideas can be put into your own scheme.

Have either of you tried this under 6.7?  The file transfer in 6.7 should 
handle this.

-Nick

-- 
           <<< Follow the white rabbit. >>>
 /`-_    Nicholas R. LeRoy               The Condor Project
{     }/ http://www.cs.wisc.edu/~nleroy  http://www.cs.wisc.edu/condor
 \    /  nleroy@xxxxxxxxxxx              The University of Wisconsin
 |_*_|   608-265-5761                    Department of Computer Sciences