[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] condor commands taking long time



That confirms that the time is spent parsing the config, or at least during early initialization.  Its most likely that the time is being spent trying to determine the IP_ADDRESS or HOSTNAME config value.

 

You can get condor_config_val to log this by configuring TOOL_DEBUG then running condor_config_val with the -debug flag. One way to do this is

 

_condor_TOOL_DEBUG=D_CAT,D_HOSTNAME,D_CONFIG,D_SUB_SECOND   condor_config_val -debug -config

 

-tj

 

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Nagaraj Panyam
Sent: Tuesday, April 26, 2022 1:22 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] condor commands taking long time

 

Hi,

 

I checked the time of codor_config_val -config. Used -verbose, too. But no useful info from it. How do I figure it out, apart from a fresh install.

It took 1m 28s but no info about what exactly is the cause.

 

[pn@sim01 ~]$ [pn@sim01 ~]$ time condor_config_val -verbose -config
Configuration source:
        /etc/condor/condor_config
Local configuration sources:
        /etc/condor/config.d/49-common
        /etc/condor/config.d/50-security
        /etc/condor/config.d/51-role-submit
        /etc/condor/condor_config.local
real    1m28.132s
user    0m0.011s
sys     0m0.010s

 

- Nagaraj

 

On Tue, Apr 26, 2022 at 7:57 PM John M Knoeller <johnkn@xxxxxxxxxxx> wrote:

One possibility is that configuration parsing is taking a long time.

 

try timing 

 

condor_config_val -summary

 

or

 

condor_config_val -config

 

-tj

 

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Nagaraj Panyam
Sent: Tuesday, April 26, 2022 7:39 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] condor commands taking long time

 

Hi,

After a recent reboot of submit node, htcondor commands are taking very long time. On Master and Exec nodes, it is fine.Even the --version switch takes nearly 1.5min

[pn@sim01 ~]$ time condor_status --version
$CondorVersion: 8.8.15 Jul 29 2021 BuildID: 552034 PackageID: 8.8.15-1 $
$CondorPlatform: x86_64_CentOS7 $

real    1m28.053s
user    0m0.010s
sys     0m0.013s

Other commands are ok. For example,

[pn@sim01 condor]$ time wc /var/log/condor/*
     120     1060    10829 /var/log/condor/KernelTuning.log
    1235     9679    82668 /var/log/condor/MasterLog
   <snip>

   26892  1514609 10485984 /var/log/condor/XferStatsLog.old
  740503  7210435 65991899 total

real    0m0.695s
user    0m0.662s
sys     0m0.028s

I notice that the process goes to sleep:

top - 17:52:59 up 40 min,  2 users,  load average: 0.06, 0.03, 0.05
Tasks: 437 total,   1 running, 435 sleeping,   1 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us, 16.7 sy,  0.0 ni, 83.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8172968 total,  7132596 free,   562416 used,   477956 buff/cache
KiB Swap:  1019900 total,  1019900 free,        0 used.  7362288 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ P COMMAND
 1561 pn        20   0  161768   2744   1308 S   0.0  0.0   0:00.19 1 sshd: pn@pts/0
 1562 pn        20   0  120064   2420   1776 S   0.0  0.0   0:00.13 0 -bash
 8667 pn        20   0  161768   2728   1300 S   0.0  0.0   0:00.14 3 sshd: pn@pts/1
 8668 pn        20   0  119928   2372   1792 S   0.0  0.0   0:00.07 0 -bash
 9536 pn        20   0  166776   2760   1704 T   0.0  0.0   0:00.15 0 top
 9564 pn        20   0   45752   4408   3732 S   0.0  0.1   0:00.00 2 condor_status --version
 9579 pn        20   0  166624   2644   1696 R   0.0  0.0   0:00.02 2 top -c -b -n 1 -u pn

 

What could be the cause?

- Nagaraj

 

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/