[Condor-users] File transfer queue - new development condor release

["David Roberts" <David.Roberts@xxxxxxxxx>]

Dear list,

My query is with regards to the new development release of condor that I believe may solve some of my current performance issues.

My Problem
When more than 30 jobs are running on my condor system they take a very long time to finish * they are not processing but are transferring files either to or from the execute node.  If I only run 20 then there is no problem i.e. the transfer speed DOES NOT scale with the number of running jobs. I assume therefore that I am getting congestion (and hence collisions?) during the many file transfers.  The problem is likely exasperated by the setup I have : The submitting machine phrtmcx00 returns all simulation data to an NFS mount to a terabyte NAS store (physnas).  Bear in mind that each of my simulations may upload between 100-400MB of data and return between 100-400MB of data.  The network is however set to full-duplex gigabit.

The solution?
The development release has the option to queue the file transfers in each direction * This should reduce the network traffic and hence fix my problem!

The questions: 
Is this the correct way to fix my problem?
If it is: Can I just install the developmental release on the submitting machine and it will queue the file transfers or do I have to install the new release on the central manager and every other machine?  I am only submitting Vanilla jobs and the system is made up of approx 10 windows and 90 linux machines.


Thanks in advance
Dave


David Roberts

Radiotherapy PhD Student
Joint Physics Department
Institute of Cancer Research and Royal Marsden NHS Trust
Downs Road,
Sutton, Surrey
UK, SM2 5PT
Tel : 020 8661 3490


The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.

This e-mail message is confidential and for use by the addressee only.  If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.