[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Preserving Data Locality in DAGs?


Can we start some discussion on this issue?

I have several multiple-core machines that end up wasting a lot of time/CPU utilization by ferrying large files between them, and it would be incredibly advantageous to have some kind of feature that allows be to group DAG nodes together so that they have a high preference to run within the same pool of VM's on the same machine.

Would anyone else using Condor DAGs find this feature useful? Is there already a configuration parameter that spans the spectrum of data locality, and I am overlooking it? I'd be curious to see others' use cases of Condor DAGs, too.


 - Armen

Armen Babikyan wrote:

Is there a way I can tell Condor to have a strong preference for running a DAG node on the same machine that the previous DAG node ran on? I'd like to do this to preserve data locality as best I can.


  - Armen

Armen Babikyan
MIT Lincoln Laboratory
armenb@xxxxxxxxxx . 781-981-1796