Re: [HTCondor-users] condor job wrapper?

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 7/1/2020 11:30 AM, Nagaraj Panyam wrote:
> Hi,
>
> Prior to executing the actual executable, we need to execute shell scripts. I have the job wrapper shell script as the 
> main executable. Within the shell I execute the shell scripts, then call the executable with its arguments
>
> The job script transfers the initial shell scripts and then the executable. However, the error is that the file 
> "myexecutable not found" even though "ls -l" shows it in the directory on the target machine. What is the correct 
> procedure for this kind of job?

The below is basically fine.

Without seeing the exact output from your ls test, or the exact error messages, here are some quick guesses/questions to 
help you troubleshoot:

1. What kind of file is myexecutable?  Is it a script, and if so, what is the first line?  Perhaps the 'file not found' 
error is really originating from the script interpreter not found on the target machine.  For instance, perhaps the 
first line of myexecutable is
  #!/usr/bin/python3
and the target machine does not have a file /usr/bin/python3...  Or perhaps you copied myexecutable from a Windows 
machine and it has incorrect line endings (which could cause similar problems).  Or perhaps the interpreter is not a 
fully qualified path.

2. How did you confirm the files are in the directory, and which directory on the target machine are we talking about? 
The job scratch directory?  Suggest you edit your jobwrapper.sh something like so -- note we output the current working 
directory:
  #!/bin/bash
  echo Debug Info
  hostname
  pwd
  ls -l
  echo End of Debug Info
  source env1.sh
  source env2.sh
  ./myexecutable arg1.txt arg2.txt

3. The default value for should_transfer_files in a submit file is IF_NEEDED, meaning if your job lands on an execute 
machine that is configured to be in the same filesystem domain as your submit machine, then transfer_input_files is 
ignored (because HTCondor assumes you are submitting your jobs from a shared filesystem).  Perhaps try adding
   should_transfer_files = YES
to your job's submit description file.

4. Does myexecutable have the execute bit set?

5. Maybe try doing "condor_submit -i <your submit file>", which should give you drop you into an interactive shell 
prompt in the environment of your job on the execute machine after your files are transferred, at which point your can 
try running ./jobwrapper.sh yourself and see if you discover anything...

regards
Todd


> ---
>
> my jobwrapper.sh looks like:
>
>     #!/bin/bash
>
>     source env1.sh
>
>     source env2.sh
>
>     ./myexecutable arg1.txt arg2.txt
>
>
> my job file looks like
>
>     transfer_input_files: env1.sh, env2.sh, myexecutable, arg1.txt, arg2.txt
>
>     executable = jobwrapper.sh
>
>
> Thanks
>
> Nagaraj
>
>
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/


--
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685