Re: [HTCondor-users] Problems with condor_ssh_to_job

[Tim Theisen <tim@xxxxxxxxxxx>]

Hi Oliver,

You hit on the correct solution. We run a script before the master
starts to ensure correct configuration. I plan to add checks (and
automatic corrections) for situations like this.

...Tim


On 11/03/2017 07:24 AM, Oliver Freyermuth wrote:
> Dear Tim and fellow users,
>
> I managed to understand the issue myself. 
>
> The problem is that our /pool directory is on a separate partition, and not in "/var/lib/condor/pool".
> Hence, our "/pool/condor" is only equipped with default_t context. 
> Since that's the default location as configured by: 
> https://github.com/HEP-Puppet/htcondor
> we thought this would be taken care of. 
>
> Now, manually executing:
> semanage fcontext -a -t condor_var_lib_t "/pool/condor(/.*)?"
> followed by:
> restorecon -v /pool/condor
> is needed to actually make the fix work. 
>
> I will look into incorporating that into puppet and open a PR against the mentioned puppet module. 
>
> Cheers,
> Oliver
>
>
> Am 02.11.2017 um 17:38 schrieb Oliver Freyermuth:
>> Dear Tim,
>>
>> sorry, but I only around to actually test the fix with the release. 
>>
>> Sadly, it still does not seem to work for me. Using 8.6.7 on CentOS 7.4, in the audit.log, I get:
>>
>> type=AVC msg=audit(1509640279.152:330): avc:  denied  { write } for  pid=52177 comm="ssh-keygen" path="/pool/condor/dir_52146/.condor_ssh_to_job_1/keygen.log" dev="sda8" ino=42729489 scontext=system_u:system_r:ssh_keygen_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=file
>> type=AVC msg=audit(1509640279.152:330): avc:  denied  { write } for  pid=52177 comm="ssh-keygen" path="/pool/condor/dir_52146/.condor_ssh_to_job_1/keygen.log" dev="sda8" ino=42729489 scontext=system_u:system_r:ssh_keygen_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=file
>> type=SYSCALL msg=audit(1509640279.152:330): arch=c000003e syscall=59 success=yes exit=0 a0=cc8700 a1=cc76e0 a2=cc3430 a3=7ffcdd442700 items=0 ppid=52172 pid=52177 auid=4294967295 uid=67803 gid=513 euid=67803 suid=67803 fsuid=67803 egid=513 sgid=513 fsgid=513 tty=(none) ses=4294967295 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=system_u:system_r:ssh_keygen_t:s0 key=(null)
>> type=PROCTITLE msg=audit(1509640279.152:330): proctitle=2F7573722F62696E2F7373682D6B657967656E002D4E00002D4300002D71002D66002F706F6F6C2F636F6E646F722F6469725F35323134362F2E636F6E646F725F7373685F746F5F6A6F625F312F7373686B6579002D7400727361
>> type=AVC msg=audit(1509640279.351:331): avc:  denied  { write } for  pid=52177 comm="ssh-keygen" name=".condor_ssh_to_job_1" dev="sda8" ino=42729487 scontext=system_u:system_r:ssh_keygen_t:s0 tcontext=system_u:object_r:default_t:s0 tclass=dir
>> type=SYSCALL msg=audit(1509640279.351:331): arch=c000003e syscall=2 success=no exit=-13 a0=55b70f811da0 a1=241 a2=180 a3=0 items=0 ppid=52172 pid=52177 auid=4294967295 uid=67803 gid=513 euid=67803 suid=67803 fsuid=67803 egid=513 sgid=513 fsgid=513 tty=(none) ses=4294967295 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=system_u:system_r:ssh_keygen_t:s0 key=(null)
>>
>> condor_submit -interactive just tells me "Failed to create ssh key ..." as before,
>> and checking the selinux contexts of the /pool/dir_XXXXX directory and also /pool/dir_XXXXX/.condor_ssh_to_job/,
>> I still get:
>> system_u:object_r:default_t:s0
>> for all directories and files within. 
>>
>> Am I missing something? 
>> Condor 8.6.7 is installed from the official HTCondor repos, and I even reinstalled the full worker node to remove any potential remnants of the old installation. 
>>
>> Cheers,
>> Oliver
>>
>>
>> Am 20.10.2017 um 23:37 schrieb Tim Theisen:
>>> Hi Oliver,
>>>
>>> I have fixed up the problem with condor_ssh_to_job and SELinux. It will
>>> be in the upcoming 8.6.7 release.
>>>
>>> https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=6362
>>>
>>> ...Tim
>>>
>>>
>>> On 08/03/2017 12:24 PM, Oliver Freyermuth wrote:
>>>> Hi Todd, 
>>>>
>>>> Am 03.08.2017 um 19:02 schrieb Todd Tannenbaum:
>>>>> Yes, you can use shared_port on your submit (schedd) machine, setup your firewall to only allow port 9618, and still have condor_ssh_to_job work with CCB.  To make it all work together this way requires that you change the permissions on the directory used by shared_port like so:
>>>>>
>>>>>    chmod 1777 /var/lock/condor/daemon_sock
>>>>>
>>>>> (/var/lock/condor is the default, you may need to do "condor_config_val LOCK" to get the actual path HTCondor lock directory).
>>>>>
>>>>> To understand why this is needed and why it works, read the wisdom for knob DAEMON_SOCKET_DIR in the HTCondor Manual, available at URL http://bit.ly/2u4Jt1F
>>>> Wow - thanks for that link, and also for the quick reply!
>>>> I totally missed that in the documentation, I was looking around in the more general parts (CCB and network configuration, and condor_ssh_to_job options and manpage),
>>>> but never checked there. Really good news we can keep the firewall up with just this (known and needed) hole for usage by HTCondor. 
>>>>
>>>>>> 2) SELinux policies prevent running ssh-keygen on the startd machine. SELinux denies permission to write the generated keys to /pool/condor/dir_<PID>/.condor_ssh_to_job_1/ .
>>>>>>     Is this already fixed in a new version of HTCondor?
>>>>>>     This breaks on CentOS 7 out of the box.
>>>>>>
>>>>> Ugh.  Thank you for investigating and reporting this. We will make a ticket on the wiki.htcondor.org to address this for the next release, and include a work-around in the ticket until the new version is released.   Tim T will post the ticket URL here shortly.
>>>> Thanks for taking care of that! 
>>>>
>>>> Best regards, 
>>>> Oliver
>>>>
>>>>
>>
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
>

-- 
Tim Theisen
Release Manager
HTCondor & Open Science Grid
Center for High Throughput Computing
Department of Computer Sciences
University of Wisconsin - Madison
4261 Computer Sciences and Statistics
1210 W Dayton St
Madison, WI 53706-1685
+1 608 265 5736