Re: [HTCondor-users] File transfer plugins & stdout/err

[Ben Jones <ben.dylan.jones@xxxxxxx>]

Hi Todd,

>> I noticed whilst writing a plugin that naming for stdout/err seems to be ignored when using âouput_destinationâ, and confirmed the behaviour with the builtin http plugin. Is there a way to get stdout/err transferred without it keeping the sandbox _condor_stdout/_condor_stderr names? Iâve tried with various transfer_output_remaps, but a solution evades me. This is with 9.0.9 on the exec node.
> 
> 	This isn't a good general solution, but when you tried transfer_out_remaps, did you disable output_destination?  Also, what happens if you specify the target URL directly in the output or error
> directive?


Aha, indeed if I just do the remaps and not the output_destination, then the remaps are honoured, ie:

executable     = yf.sh
arguments      = $(ClusterId) $(ProcId)
log            = xfer.$(ClusterId).log
error          = yf.$(ClusterId).$(ProcId).err
output         = yf.$(ClusterId).$(ProcId).out
transfer_output_remaps = "_condor_stdout = http://bsub.cern.ch:44444/yf.$(ClusterId).$(ProcId).out; _condor_stderr = http://bsub.cern.ch:44444/yf.$(ClusterId).$(ProcId).err"
# output_destination = root://eosuser.cern.ch//eos/user/b/bejones/condor/xfer/
transfer_input_files = root://eosuser.cern.ch//eos/user/b/bejones/condor/file.txt
requirements = OpSysAndVer =?= "CentOS8"
queue

On the output endpoint:

[root@bsub ~]# ./ws.py
serving at port 44444
128.142.130.177 - - [03/Feb/2022 20:15:24] "PUT /yf.77.0.err HTTP/1.1" 201 -
128.142.130.177 - - [03/Feb/2022 20:15:25] "PUT /yf.77.0.out HTTP/1.1" 201 -

But yeah, on its own this is more than a little clunky (albeit one I guess I could hack with a schedd xform).

The other suggestion, to specify the target URL directly for stdout/err, the behaviour changes depending on whether or not I have output_destination set. If output_destination is defined, then stdout/err go there as _condor_stdout/err, and use whatever plugin the output_destination is using, ignoring the method for Out/Err as well as the filename.

If I donât set an output_destination, then the exe tries to send the out/err back to the schedd, but it does not go well, ie:

executable     = yf.sh
arguments      = $(ClusterId) $(ProcId)
log            = xfer.$(ClusterId).log
error          = http://bsub.cern.ch:44444/yf.$(ClusterId).$(ProcId).err
output         = http://bsub.cern.ch:44444/yf.$(ClusterId).$(ProcId).out
requirements = OpSysAndVer =?= "CentOS8"
queue

$ condor_q 80.0 -af HoldReason
Error from slot1_1@xxxxxxxxxxxxxxxxxx: STARTER at 128.142.130.177 failed to send file(s) to <188.185.87.2:9618>; Remap of output file resulted in a URL: http://bsub.cern.ch:44444/yf.80.0.out

cheers,
Ben