Re: [HTCondor-users] Tracking resource utilization in condor_submit -interactive

[Ian McEwen <mian@xxxxxxxxxxx>]

I can't speak to whether the configuration is at fault, but a detail
which may be useful for testing (which you may know, but many don't --
my apologies if I'm saying something you already know): the contents of
/proc/self/cgroup should tell you what cgroups the current process is
running in, which should help confirm if things are ending up in the
cgroups you'd expect.

Hope this helps!

--
Ian

On Tue, Feb 13, 2018 at 10:27:13PM +0000, Michael Pelletier wrote:
> Yeah, I've been using cgroups since the early days, and I think I have at least one or two cgroup-related bugs in the database. ;)
> 
> So an example here is that one of our users is running a Xilinx Vivado session to do some interactive simulation and debugging of an FPGA design.  The Vivado executable in his interactive session shows 7:52 worth of CPU time in the ps output column, and there's maybe one extra minute of  time in the adjunct processes.  The "top" output shows the Vivado session at 1.624 gigabytes of resident memory, and a 7.1GB virtual image size.
> 
> However:
> 
> [pelletm ~]$ condor_q 4558 -af '(remotesyscpu + remoteusercpu)' memoryusage
> 0.0 undefined
> [pelletm  ~]$
> 
> Do I maybe have the cgroups configured incorrectly for CentOS/RHEL 7? My BASE_CGROUP config is set to htcondor, which dates back to version 8.0 or whenever that was introduced. Do I need to change that now that cgroups are woven into the kernel and systemd? Maybe the hierarchy is not getting picked up properly?
> 
> Thanks!
> 
> 	-Michael Pelletier.
> 
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Greg Thain
> Sent: Tuesday, February 13, 2018 4:28 PM
> To: htcondor-users@xxxxxxxxxxx
> Subject: [External] Re: [HTCondor-users] Tracking resource utilization in condor_submit -interactive
> 
> On 02/05/2018 12:24 PM, Michael Pelletier wrote:
> > We've got a few folks who need interactive shells due to the nature of 
> > their workflow, and I've set up appropriate submit descriptions for 
> > use with condor_submit -interactive. However, the job attributes such 
> > as MemoryUsage and RemoteUserCpu don't track the resource utilization 
> > by the condor_ssh_to_job sessions launched by condor_submit (as would 
> > be desired under any other circumstance),
> 
> Do you have cgroups enabled? With cgroups, the ssh session is placed in a child cgroup of the main job, so the usage should show up in the sleep job.
> 
> -greg
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/