Re: [HTCondor-users] CPU usage & network usage not reported correctly with recent versions of Docker

[andrew.lahiff@xxxxxxxxxx]

I should add that after adding:

STARTER_DEBUG = D_FULLDEBUG

I can see from the StarterLog that sys cpu seems to be read correctly from the docker socket [1], but despite this RemoteSysCpu still ends up being zero (this example is from HTCondor 8.6.1).

Regards,
Andrew.

[1]
# grep "docker stats reports" /var/log/condor/StarterLog
03/05/17 13:03:12 docker stats reports max_usage is 2347008 rx_bytes is 508 tx_bytes is 508 usage_in_usermode is 370000000 usage_in-sysmode is 1320000000
03/05/17 13:03:34 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 4860000000 usage_in-sysmode is 18760000000
03/05/17 13:03:56 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 9360000000 usage_in-sysmode is 36170000000
03/05/17 13:04:18 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 13820000000 usage_in-sysmode is 53590000000
03/05/17 13:04:40 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 18230000000 usage_in-sysmode is 71120000000
03/05/17 13:05:02 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 22740000000 usage_in-sysmode is 88530000000
03/05/17 13:05:24 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 27300000000 usage_in-sysmode is 105910000000
03/05/17 13:05:46 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 31780000000 usage_in-sysmode is 123370000000
03/05/17 13:06:08 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 36300000000 usage_in-sysmode is 140770000000
03/05/17 13:06:30 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 40850000000 usage_in-sysmode is 158140000000
03/05/17 13:06:52 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 45310000000 usage_in-sysmode is 175620000000
03/05/17 13:07:14 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 49860000000 usage_in-sysmode is 192970000000
03/05/17 13:07:36 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 54400000000 usage_in-sysmode is 210350000000
03/05/17 13:07:58 docker stats reports max_usage is 2347008 rx_bytes is 648 tx_bytes is 648 usage_in_usermode is 58890000000 usage_in-sysmode is 227810000000


________________________________________
From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of andrew.lahiff@xxxxxxxxxx [andrew.lahiff@xxxxxxxxxx]
Sent: Sunday, March 05, 2017 10:54 AM
To: htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] CPU usage & network usage not reported correctly with recent versions of Docker

Hi Greg,

I can confirm that when HTCondor 8.6.1 is used with the latest version of Docker (17.03.0), RemoteUserCpu, NetworkInputMb and NetworkOutputMb seem to be correct.

However, I noticed that RemoteSysCpu is always zero [1]. RemoteSysCpu is also always zero when HTCondor 8.6.0 is used with older versions of Docker, e.g. 1.12.6 [2]. Are there any combinations of HTCondor and Docker versions where RemoteSysCpu is known to be reported correctly?

Thanks,
Andrew.

[1] it's easy to write a little test code that results in non-zero sys cpu time.
[2] both with real jobs and a test code.

________________________________________
From: Lahiff, Andrew (STFC,RAL,SC)
Sent: Thursday, February 16, 2017 6:07 PM
To: HTCondor-Users Mail List
Subject: RE: [HTCondor-users] CPU usage & network usage not reported correctly with recent versions of Docker

Hi Greg,

Thanks for the quick reply, I'll try 8.6.1 when it comes out.

Regards,
Andrew.

________________________________________
From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of Greg Thain [gthain@xxxxxxxxxxx]
Sent: Thursday, February 16, 2017 5:08 PM
To: htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] CPU usage & network usage not reported correctly with recent versions of Docker

On 02/16/2017 10:34 AM, andrew.lahiff@xxxxxxxxxx wrote:
> Hi,
>
> I noticed that with Docker 1.13.1 HTCondor seems to no longer report CPU usage correctly (RemoteUserCpu & RemoteSysCpu are always both zero) or network stats (NetworkInputMb & NetworkOutputMb don't exist at all). With older versions of Docker, e.g. 1.12.6, everything seems fine. Has anyone else found this?
>

Andrew:

We just put a fix into 8.6.1 for something very similar, HTCondor ticket
#6088, which I think will fix this problem.

-greg
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/