Re: [HTCondor-users] Question about th hold reason "errno 8 (Exec format error)"

[Greg Thain <gthain@xxxxxxxxxxx>]

On 9/25/21 8:12 PM, jiangxw@xxxxxxxxxx wrote:

Dear all,

 

These days I got a hold problem in my glidein cluster as the subject described “errno 8 (Exec format error)”.

It always caused by a job script without the interpreter line, like “#!/bin/bash” and seems htcondor can detect it and hold the job.

Is there a way to disable condor to check the job script interpreter? (It’s hard to mandatory all the users have a good habit of writing their job script.)

(the condor versions I’ve tested are 8.9.7 and 9.0.6, the rpm package is condor-x86_64_CentOS7-stripped.tar.gz.)

 

Hi Xiaowei:

When HTCondor runs a job on Linux, it forks and exec's the executable that is named in the "executable" line in the submit file.  The exec system call looks at the first few bytes in the executable to determine if it is an ELF binary to run, or if it starts with the characters "#!", to run it with some interpreter, like a shell or python.  If it is none of these, then we get the errno 8 (Exec format error), and HTCondor puts the job on hold.

This is a little different than when you are running a command from the shell.  When the shell runs a program, it doesn't just pass the program name (and arguments) to the kernel to exec.  Before it does that, the shell itself, in user space, looks at the program you have asked it to run, and if it doesn't look like an executable, the shell treats the problem like a shell script, as if it started with '#!', and passes /bin/sh to the kernel.

Some other batch systems always run their executables under a shell, so that you get this behavior, but HTCondor does not, by default.

If you want all of your HTCondor jobs to run under a shell, and have this functionality (and maybe some other shell-like functions), you can force all HTCondor jobs to be started by a shell with a user job wrapper -- just set on the worker node:

USER_JOB_WRAPPER = /path/to/script.sh

where /path/to/script.sh just contains

#!/bin/sh

exec "$@"

And I think that shell scripts missing the interpreter line will run under HTCondor.

-greg