Re: [HTCondor-users] File transfer plugins & stdout/err

[Ben Jones <ben.dylan.jones@xxxxxxx>]

> On 3 Feb 2022, at 22:36, Todd L Miller <tlmiller@xxxxxxxxxxx> wrote:
> 
>> Aha, indeed if I just do the remaps and not the output_destination, then the remaps are honoured, ie:
> 
> ...
> 
>> But yeah, on its own this is more than a little clunky (albeit one I guess I could hack with a schedd xform).
> 
> 	OK, with this combination of results, I think I know where the problem is and might be able to fix it in relatively short order.  One question though: your example specifies a basename for output and error, which makes it easy to determine what the URL should be:
> f"{output_destination}/{output}".  If the value of output or error
> includes path, what should the URL look like?  Does the answer change if the value is an absolute or relative path?

This is a good questionâ

The use case for us is this - our bog-standard user workflow is: remote submit with shared fs between remote submit machine and schedd, so normally weâd expect relative paths. Weâre trying to wean people off the shared fs, but more or less preserve the workflow.

In the case of a relative path, say with `output = output/out.txt` Iâd expect to try f"{url}/output/out.txt" and fail if the dir doesnât exist. With absolute paths I donât know if thereâs a good answer. I think Iâd be tempted to say it should either try f"{url}/{full_path}â or just fail at submit.

BTW, whilst weâre on slightly weird things that happen with output_destination, if I do a condor_submit -spool, then you get the same behaviour with stdout/err, but also when you try condor_transfer_data, it tries to use the plugin on the schedd to post the UserLog to output_destination too.


> 	In the meantime, if you weren't aware, the job ClassAd is available in the job sandbox as `.job.ad`, and your plug-in should be able to read it and determine what the special cases of "_condor_stdout" and
> "_condor_stderr" should be named.

Oh, good point, I hadnât thought to just use the .job.ad. Great tip, thanks.

>> If I donât set an output_destination, then the exe tries to send the out/err back to the schedd, but it does not go well, ie:
> 
> 	I'm sure we had a good reason for preventing that from working, but I don't remember what it is. ;)

On a similar note is there a reason why executable canât be URL ?

cheers,
Ben