Re: [Condor-users] heavy loads

[Steven Timm <timm@xxxxxxxx>]

There are two types of globus-job-manager processes using
the globus which condor refers to as "gt2".

One is a jobmanager-condor script which stays live as long as the
job is live.

The second is a globus-job-manager-script perl script which runs
once per minute and is forked off from the main jobmanager-condor.

Do pstree, I think you will see several others are children of the one.
I've seen this happen in cases where there is some problem with
NFS and the globus-job-manager is stuck trying to delete or undelete
a hard link across NFS.  strace will tell you if this is the case.
Oftentimes it can be just one process that is hung waiting for some
nfs file and once you kill that process the rest of them will clear out.

As to why there is at least one globus-job-manager per condor job,
there are several reasons why it could be.  Did you disable streaming?
You have to for the grid monitor to work.  Do you see
jobmanager-forks trying to start from your client node to your head node?
Those are needed to start up the grid monitor.  IS ENABLE_GRID_MONITOR
set to TRUE in your condor_config file?  It needs to be.

In the archives of this list there is a procedure to try to start
up the grid monitor manually, to see if you have any problems
that are blocking the automatic condor-G start.

Steve


------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525  timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Div/Core Support Services Dept./Scientific Computing Section
Assistant Group Leader, Farms and Clustered Systems Group
Lead of Computing Farms Team

On Thu, 29 Jun 2006, Michael Thomas wrote:

> I forgot to mention that this is using condor 6.7.18, and there are > 1300 
> jobs in the queue right now (all but 200 are idle).
>
> --Mike
>
> Michael Thomas wrote:
> > While doing some stress testing on our 200-node cluster using condor-g, we 
> > have noticed some extremely large loads on the cluster.  The large load 
> > seems to be caused by 500+ globus-job-manager processes, with sometimes 2 
> > or 3 globus-job-manager processes for each job.
> >
> > condor_config contains the line:
> > GRIDMANAGER_MAX_JOBMANAGERS_PER_RESOURCE = 10
> > ...but that seems to be ignored.
> >
> > Why would we have multiple globus-job-managers for a single job, and what 
> > can we do to reduce the number of globus-job-manager processes so that our 
> > gatekeeper doesn't get quite so overloaded?
> >
> > --Mike
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Condor-users mailing list
> > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> >
> > The archives can be found at either
> > https://lists.cs.wisc.edu/archive/condor-users/
> > http://www.opencondor.org/spaces/viewmailarchive.action?key=CONDOR