[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] transfer_input_files, arguments......executable not transferred

Debugging the problem deeper it seems that the executable gets transferred only if it is defined in the "transfer_input_files" list.....

So given a

queue transfer_input_files , arguments from [0:1:1] ./filelist

kind of queue statement, what format should i use in the filelist file to be bale to list multiple input files (the executable and data files to process) and multiple arguments ?

thanks a lot,

From: John M Knoeller <johnkn@xxxxxxxxxxx>
Sent: Thursday, April 4, 2019 6:49 PM
To: HTCondor-Users Mail List
Cc: Gergely Debreczeni
Subject: RE: [HTCondor-users] transfer_input_files, arguments......executable not transferred

For historical reasons, HTCondor will rename your executable to condor_exec.exe when it transfers it.  so if you want to know if the executable is transferred, you need to stat condor_exec.exe.  


Also, there is no need to use ./ before your executable name.   This should work fine


## Executable

executable              = sdfgen.sh

universe                = docker

docker_image            = dani_tensorflow:v_01


At some time in the 8.9 series, we plan to stop renaming the executable, but it is actually a lot of work to remove this behavior cleanly.




From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Gergely Debreczeni via HTCondor-users
Sent: Thursday, April 4, 2019 8:21 AM
To: htcondor-users@xxxxxxxxxxx
Cc: Gergely Debreczeni <gergely.debreczeni@xxxxxxxxxxxx>
Subject: [HTCondor-users] transfer_input_files, arguments......executable not transferred


Hi !


I have a single executable (sdfgen.sh) that I'd like to run in thousands of times with different input files and parameters. I found the submit file example below, where one lists the input files and parameters in a text file and gives that as a parameter to the Queue "command".


Everything seems to work fine, except that the executable itself is not transferred, even if I add the transfer_executable=true line into my sub file.


The submitted jobs get started apparently with the right parameter set but are all put in 'Hold' with the following message:



044.000:  Request is held.


Hold reason: Error from slot1@scorpio005: Error running docker job: OCI runtime create failed: container_linux.go:344: starting container process caused 'exec: \'./sdfgen.sh\': stat ./sdfgen.sh: no such file or directory': unknown



How I can resolve this issue ?


Thanks in advance,







The sub fiile:


## Executable

executable              = ./sdfgen.sh

universe                = docker

docker_image            = dani_tensorflow:v_01


## Logs

log                     = out/sdfgen.$(Process).log

output                  = out/sdfgen.$(Process).stdout

error                   = out/sdfgen.$(Process).stderr


## File transfer

transfer_executable     = true

should_transfer_files   = Yes

when_to_transfer_output = ON_EXIT

line = $(Row)+1


transfer_output_files   = output.out

transfer_output_remaps  = "output.out=out/output$INT(line).out"



## Resources requested

request_cpus            = 1

request_GPUs            = 1

Requirements            = (ResourceType == "Dedicated") && (regexp(".*dani_tensorflow.*",LocallyAvailableDockerImages))



## Submit command

queue transfer_input_files,arguments from [0:3:1] ./filelist





The filelist file:



a, a 14.3 4

b, b 2.3 3

c, c 23.4 1



This e-mail and any files transmitted with it contain confidential and may contain privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized use, copying, disclosure or distribution of the material in this e-mail is strictly forbidden.