Re: [Condor-users] DAGMan memory consumption

[Ian Stokes-Rees <ijstokes@xxxxxxxxxxxxxxxxxxx>]

On 6/24/10 4:06 PM, Ian Stokes-Rees wrote:

It appears that DAGMan consumes 200 MB per DAG that we submit, regardless of DAG size.  This obviously introduces some limitations on the number of concurrent DAG jobs we can submit.  Is there anything we can do to reduce the memory consumption per-DAG?  Is it on the near-term roadmap to profile and optimize condor_dagman to reduce this, if there aren't things I can do?

I lied.  I was looking at the wrong condor_dagman processes -- a small DAG has a small memory footprint.

Can we help ourselves by having our 100k node DAG divided into, say, 100 sub-DAGs of 1000 nodes each, and have the dependencies setup so 10-20 of these are active at any time?

TIA for advice on DAG construction and condor_dagman management.

Ian