[HTCondor-users] SSH/RSync clients not working in Condor job due to SELinux

Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab

Hi,

following the previous thread about the SSH-in-job issue, we have found the reason and hope the package maintainer can fix it in future releases.

##Problem

SSH/RSync clients fail to start in any Condor job running on a SELinux enabled execute node.

##Operating system

CentOS 7

##Cause of the problem

The following SELinux rule is set in the OS: `type_transition condor_startd_t ssh_exec_t : process condor_startd_ssh_t;`, so the SSH client process is given the context domain `condor_startd_ssh_t` which has NO access to $HOME, /dev/tty or /dev/pts/*. Therefore the SSH command fails silently.

##Why SSH matters

In our MPI setup for ABAQUS, SSH command is used to start remote session on machines.

Best regards,

Mingxuan

Von: "Lukas Koschmieder" <Lukas.Koschmieder@xxxxxxxxxxxxxxxxxxx>
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
Gesendet: Freitag, 26. Oktober 2018 15:15:57
Betreff: Re: [HTCondor-users] ssh/scp returning exit code 255 in Bash jobs

Hi,

- Do really want your job to deal with scripting to use scp and passing
around credentials? I think usually it is better/easier/safer to use
HTCondor's file transfer mechanism (which is scheduled) to move files
from your submit machine to the execute machine

I use Condor to spawn JupyterLab servers on remote machines. So my goal is to enable actual human users to run ssh, scp and condor_ssh_to_job on their JupyterLab servers.

- If you need to use scp etc, the most common reason I've encountered
for an exit status of 255 in scripts that run outside of my home
environment is because I am contacting a host not in my
$HOME/.ssh/known_hosts file (almost certainly not the case if you are
using HTCondor without a shared file system or across administrative
domains). To remedy, consider passing command-line switches to scp/ssh
like so:
scp -o UserKnownHostsFile=./my_temporary_known_host ssh_host ...

$HOME/.ssh/know_hosts is definitely available.

Could this problem be somehow related to TTY?

Here are the results of a recent test:

Error file:

condor_exec.exe: cannot set terminal process group (-1): Inappropriate ioctl for device
condor_exec.exe: no job control in this shell
unknown option -- h
usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
           [-D [bind_address:]port] [-E log_file] [-e escape_char]
           [-F configfile] [-I pkcs11] [-i identity_file]
           [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           [user@]hostname [command]

Job description file:

universe = vanilla
executable = /bin/bash
transfer_executable = false
arguments =" -li -c 'ssh -h' "
queue

Cheers,

Lukas

--
Lukas Koschmieder
Steel Institute IEHK
RWTH Aachen University
Intzestraße 1
52072 Aachen
Germany

Tel: +49 (0)241 80 95823
Fax: +49 (0)241 80 92253
lukas.koschmieder@xxxxxxxxxxxxxxxxxxx

From: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
Sent: Tuesday, October 16, 2018 10:49:16 PM
To: HTCondor-Users Mail List; Koschmieder, Lukas
Subject: Re: [HTCondor-users] ssh/scp returning exit code 255 in Bash jobs

On 10/16/2018 9:51 AM, Koschmieder, Lukas wrote:
> Hi,
>
>
> When I try to run ssh/scp in a Condor Bash job, the command always fails
> with exit code 255. Are you familiar with this behavior? How do I enable
> ssh/scp in my Condor jobs?
>
>
> Cheers,
>
> Lukas
>

Couple quick thoughts:

- Do really want your job to deal with scripting to use scp and passing
around credentials? I think usually it is better/easier/safer to use
HTCondor's file transfer mechanism (which is scheduled) to move files
from your submit machine to the execute machine ... see
http://tinyurl.com/ya69km5j

- If you need to use scp etc, the most common reason I've encountered
for an exit status of 255 in scripts that run outside of my home
environment is because I am contacting a host not in my
$HOME/.ssh/known_hosts file (almost certainly not the case if you are
using HTCondor without a shared file system or across administrative
domains). To remedy, consider passing command-line switches to scp/ssh
like so:
scp -o UserKnownHostsFile=./my_temporary_known_host ssh_host ...

Hope the above helps
Todd

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx

Hi,