[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Condor-users] File transfer queue - new development condor release
- Date: Thu, 29 Nov 2007 19:33:20 +0000
- From: "David Roberts" <David.Roberts@xxxxxxxxx>
- Subject: [Condor-users] File transfer queue - new development condor release
My query is with regards to the new development release of condor that I believe may solve some of my current performance issues.
When more than 30 jobs are running on my condor system they take a very long time to finish * they are not processing but are transferring files either to or from the execute node. If I only run 20 then there is no problem i.e. the transfer speed DOES NOT scale with the number of running jobs. I assume therefore that I am getting congestion (and hence collisions?) during the many file transfers. The problem is likely exasperated by the setup I have : The submitting machine phrtmcx00 returns all simulation data to an NFS mount to a terabyte NAS store (physnas). Bear in mind that each of my simulations may upload between 100-400MB of data and return between 100-400MB of data. The network is however set to full-duplex gigabit.
The development release has the option to queue the file transfers in each direction * This should reduce the network traffic and hence fix my problem!
Is this the correct way to fix my problem?
If it is: Can I just install the developmental release on the submitting machine and it will queue the file transfers or do I have to install the new release on the central manager and every other machine? I am only submitting Vanilla jobs and the system is made up of approx 10 windows and 90 linux machines.
Thanks in advance
Radiotherapy PhD Student
Joint Physics Department
Institute of Cancer Research and Royal Marsden NHS Trust
UK, SM2 5PT
Tel : 020 8661 3490
The Institute of Cancer Research: Royal Cancer Hospital, a charitable Company Limited by Guarantee, Registered in England under Company No. 534147 with its Registered Office at 123 Old Brompton Road, London SW7 3RP.
This e-mail message is confidential and for use by the addressee only. If the message is received by anyone other than the addressee, please return the message to the sender by replying to it and then delete the message from your computer and network.