Re: [HTCondor-users] Input/Output file transfers & shared filesystems

[Petr Vokac <petr.vokac@xxxxxxx>]

I would like to configure local batch to be more simple for our users
while at same time I would like to avoid overloading schedd by
sub-optimal user decisions in their job submission files by htcondor
configuration.

Transfer configuration `IF_NEEDED` works except in a case when user adds
`use_x509userproxy` in the job submission file. Proxy is by default
stored in /tmp and not on shared filesystem and user must also specify
`should_transfer_files = YES`. Also it seems to me unnecessary
complicated to explain users why `transfer_input_files` and
`transfer_output_files` works fine for normal jobs but cause troubles
with jobs that use X509 proxy / enforced job transfers.

I think `FILESYSTEM_DOMAIN` mechanism is not flexible enough, because in
practice you don't share whole filesystem between submission and worker
node. Actually list of shared paths is quite limited (e.g. /home, /mnt,
/net) and even much more simple batch implementation provide mechanism
to configure this list of shared filesystems.

Petr

On 5/12/21 10:44 PM, Mark Coatsworth wrote:
> Hi Petr,
>
> When you say "automatically transfer files from non-shared areas" are
> you expecting it to look automatically in certain areas? Currently
> there's no way to do this exclusive to file transfer. You need to use
> the following:
>
> transfer_input_files = /tmp/file1, /tmp/file2, ...
>
> You could use the initialdir submit command, but then condor will
> expect all your other files (executable, output, log, etc.) to also be
> relative to that directory:
>
> initialdir = /tmp
> transfer_input_files = file1, file2
>
> There's no way to exclude directories from the file transfer list.
> Although I'm not sure why you need to? If the worker nodes have access
> to the shared FS, you can just assume that any files you need will
> already be there, you don't need to add them to the input file list.
>
> Can you explain more what you're trying to do here?
>
> Mark
>
> On Mon, May 10, 2021 at 12:17 PM Petr Vokac <petr.vokac@xxxxxxx> wrote:
>> Hi,
>>
>> how to configure HTCondor to automatically transfer files from
>> non-shared areas (e.g. /tmp) while at same time not to use condor
>> transfers for files from shared filesystems (e.g. /mnt, /home, ...)
>> reachable from WN?
>>
>> Petr
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>
>