Re: [HTCondor-users] ShadowLog not accessible for Shadows, jobs not starting/not ending up in the right slot for their Shadow

[Michael Pelletier <michael.v.pelletier@xxxxxxxxxxxx>]

The bug is not always harmless.

When a system is running auditd to capture failed operations on security-relevant elements of the system, the /var/log directory is under auditd scrutiny. When ShadowLog is located in its default /var/log/condor, this means that a job-owner attempt to update the ctime is logged as an audit event. This results in a steady cadence of false-positive audit events making it look like every HTCondor user is trying to tamper with the /var/log directory, thus inflating the size of the audit log, wasting disk space for the duration of the audit log retention period, and annoying the security officer responsible for reviewing it.

I guess I should have sent in a bug report a few years ago, sorry! I worked around it by configuring the ShadowLog to a directory, /dev/shm, which wasn't being watched by auditd.


Michael V. Pelletier
Information Technology
Digital Transformation & Innovation
Integrated Defense Systems
Raytheon Company

-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Jaime Frey
Sent: Wednesday, February 5, 2020 3:35 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [External] Re: [HTCondor-users] ShadowLog not accessible for Shadows, jobs not starting/not ending up in the right slot for their Shadow

> On Feb 5, 2020, at 2:10 PM, Todd L Miller <tlmiller@xxxxxxxxxxx> wrote:
> 
>> Such a shadow process tries to open the ShadowLog but seem to fail 
>> with a 'Operation not permitted' [2].
> 
> 	In the log fragment referenced here, the only EPERM I see is for chmod().  The return code of the chmod() I see in the logging code is deliberately ignored, so I'd be surprised if this was directly responsible for a problem.  It does, however, seem a little strange.


The EPERM error for chmod() is a small bug in the logging code thatâs harmless. Every daemon updates the ctime of its log file once a minute. This lets us approximate the time at which a daemon locks up or crashes, if it wasnât writing to the log at the time. The shadow is attempting to do this periodic ctime update as the wrong user (as the job owner instead of as user condor). Since multiple shadow daemons share the same log file, the ctime update isnât helpful most of the time.

 - Jaime