[HTCondor-users] condor commands taking long time

[Nagaraj Panyam <pn@xxxxxxxxxxx>]

Hi,

After a recent reboot of submit node, htcondor commands are taking very long time. On Master and Exec nodes, it is fine.Even the --version switch takes nearly 1.5min

[pn@sim01 ~]$ time condor_status --version
$CondorVersion: 8.8.15 Jul 29 2021 BuildID: 552034 PackageID: 8.8.15-1 $
$CondorPlatform: x86_64_CentOS7 $

realÂÂÂ 1m28.053s
userÂÂÂ 0m0.010s
sysÂÂÂÂ 0m0.013s

Other commands are ok. For example,

[pn@sim01 condor]$ time wc /var/log/condor/*
 Â 120 1060 10829 /var/log/condor/KernelTuning.log
  1235 9679 82668 /var/log/condor/MasterLog
ÂÂ <snip>

ÂÂ 26892Â 1514609 10485984 /var/log/condor/XferStatsLog.old
 740503 7210435 65991899 total

realÂÂÂ 0m0.695s
userÂÂÂ 0m0.662s
sysÂÂÂÂ 0m0.028s

I notice that the process goes to sleep:

top - 17:52:59 up 40 min, 2 users, load average: 0.06, 0.03, 0.05
Tasks: 437 total,ÂÂ 1 running, 435 sleeping,ÂÂ 1 stopped,ÂÂ 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 16.7 sy, 0.0 ni, 83.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8172968 total, 7132596 free, 562416 used, 477956 buff/cache
KiB Swap: 1019900 total, 1019900 free, 0 used. 7362288 avail Mem

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
Â1561 pnÂÂÂÂÂÂÂ 20ÂÂ 0Â 161768ÂÂ 2744ÂÂ 1308 SÂÂ 0.0Â 0.0ÂÂ 0:00.19 1 sshd: pn@pts/0
Â1562 pnÂÂÂÂÂÂÂ 20ÂÂ 0Â 120064ÂÂ 2420ÂÂ 1776 SÂÂ 0.0Â 0.0ÂÂ 0:00.13 0 -bash
Â8667 pnÂÂÂÂÂÂÂ 20ÂÂ 0Â 161768ÂÂ 2728ÂÂ 1300 SÂÂ 0.0Â 0.0ÂÂ 0:00.14 3 sshd: pn@pts/1
Â8668 pnÂÂÂÂÂÂÂ 20ÂÂ 0Â 119928ÂÂ 2372ÂÂ 1792 SÂÂ 0.0Â 0.0ÂÂ 0:00.07 0 -bash
Â9536 pnÂÂÂÂÂÂÂ 20ÂÂ 0Â 166776ÂÂ 2760ÂÂ 1704 TÂÂ 0.0Â 0.0ÂÂ 0:00.15 0 top
Â9564 pnÂÂÂÂÂÂÂ 20ÂÂ 0ÂÂ 45752ÂÂ 4408ÂÂ 3732 SÂÂ 0.0Â 0.1ÂÂ 0:00.00 2 condor_status --version
Â9579 pnÂÂÂÂÂÂÂ 20ÂÂ 0Â 166624ÂÂ 2644ÂÂ 1696 RÂÂ 0.0Â 0.0ÂÂ 0:00.02 2 top -c -b -n 1 -u pn

What could be the cause?

- Nagaraj