[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Most of the time in Condor jobs gets wasted in I/o
- Date: Thu, 25 Apr 2013 18:38:09 +0200
- From: "Dr. Harinder Singh Bawa" <harinder.singh.bawa@xxxxxxxxx>
- Subject: Re: [HTCondor-users] Most of the time in Condor jobs gets wasted in I/o
Hi Dimitri Maziuk,
I understand that nfs is not meant for higher data transfer. But whatever the resources I have, I can test some of the ideas I get from this thread.
I tried using "should_transfer_files = NO" but I see no difference.
Second, I see you suggested to transfer file to local area i.e /var/tmp/ and you suggested script for that too. I can certainly try this but just wondering that "cp" command will work for the data of say 150GB on one node.(15TB divided into 100 jobs/nodes). Certainly it works but take lot of time to copy. Is there any other command I can use instead of cp.which is meant to be faster
On Wed, Apr 24, 2013 at 11:28 PM, Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>
On 04/24/2013 03:36 PM, Dr. Harinder Singh Bawa wrote:One way would be to run your job as a DAG and have a PRE script that
> Idea of Transferring files to worker node also comes to my mind but I am
> not sure how to do this.
does 'for i in xargs -a file.list ; do cp /scratch/$i /var/tmp ; done'
(the job then reads from /var/tmp): see
You could also use 'transfer_input_files', but the above is probably
It only looks bad if you haven't seen it before. We tried to have BLAST
> One more thing I dont understand is that if I test to run , one such
> command interactively on one such node, it finishes in 40 minutes. The same
> command in condor doesn't finishes in more than 1 day
mmap the same set of 2GB database files over NFS. I think it only took a
couple of dozen nodes, nowhere near 120, to push overall i/o wait times
over 24 hours. It is surprising how bad nfs gets once the filesystem is
over ~75% full, too...
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
You can also unsubscribe by visiting
The archives can be found at:
Dr. Harinder Singh Bawa
Experimental High Energy Physics