[Condor-users] DAGMan memory consumption

[Ian Stokes-Rees <ijstokes@xxxxxxxxxxxxxxxxxxx>]

It appears that DAGMan consumes 200 MB per DAG that we submit, regardless of DAG size.  This obviously introduces some limitations on the number of concurrent DAG jobs we can submit.  Is there anything we can do to reduce the memory consumption per-DAG?  Is it on the near-term roadmap to profile and optimize condor_dagman to reduce this, if there aren't things I can do?

Thanks,

Ian

More details:

We just noticed this as I submitted 6 DAGs and an hour later 4 of them had close to 100% of their 200MB memory usage resident and were consuming ~5-10% CPU each.  Although these were big DAGs (100k nodes), we repeated the process with a very small DAG (40 nodes), and the condor_dagman process also consumed 200 MB.  I'm pretty sure this isn't multi-threaded, so the conclusion is that each DAG we submit will consume about 200 MB of RAM, with a good possibility that most of this will be resident (i.e. the process is busy managing the DAG jobs, and accesses all of its 200 MB image to do this).

-- 
Ian Stokes-Rees, PhD                       W: http://hkl.hms.harvard.edu
ijstokes@xxxxxxxxxxxxxxxxxxx               T: +1 617 432-5608 x75
NEBioGrid, Harvard Medical School          C: +1 617 331-5993