[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] increasing schedd memory usage [v8.6.0?]



Just as a data point, fwiw, I just looked at the ganglia chart for a fairly busy (~7000 jobs running at any moment) schedd here at UW-Madison which has been running v8.6.0 for three weeks. No sign of memory leaks or bursts.

regards,
Todd


On 2/10/2017 9:13 AM, Thomas Hartmann wrote:
Hi Brian,

thanks for the suggestion

On 2017-02-10 03:19, Brian Bockelman wrote:
Is it possible the extra memory usage is coming from when the condor_schedd process forks to respond to condor_q queries?  Are you seeing an abnormally large amount of queries?

not that I am aware of - any queries would should come only from the ARC
CE, but afais both our ARCCEs have been ~equally busy.
As cross-check, I restarted the CE daemon, but it had no effect on the
memory consumption so far and only reduced the number of connections to
the outside [1] compared to its sibling (should be the expected behaviour).
On the affected node quite(?) a number of shadows were kept open [2],
but that should be OK, or?

We have now downgraded the version to
  8.4.11
and will keep an eye on it over the weekend.
If the behaviour gets back to normal, we can at least exclude Condor.

Cheers,
  Thomas




[1]
grid-arcce1 > wc -l /proc/net/tcp*
335 /proc/net/tcp
10 /proc/net/tcp6
345 total

grid-arcce0 > wc -l /proc/net/tcp*
2733 /proc/net/tcp
16 /proc/net/tcp6
2749 total


[2]
lsof -i TCP | grep condor | cut -d " " -f 1 | sort | uniq -c
      1 condor_de
      1 condor_ma
      4 condor_sc
lsof | grep condor | cut -d " " -f 1 | sort | uniq -c
     27 condor_de
     30 condor_ma
     19 condor_pr
     45 condor_sc
  44776 condor_sh
      1 scan-cond



_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/



--
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685