Re: [HTCondor-users] HTCondor file-transfer vs networked storage

[Greg Thain <gthain@xxxxxxxxxxx>]

On 8/22/22 15:35, Matthew T West via HTCondor-users wrote:
> When working on a single homogeneous compute cluster, are there any 
> advantages to using HTCondor's file-transfer rather than working off 
> shared network storage? 


Hi Matthew:

There are several advantages to using explicit file transfer. Perhaps 
the biggest advantage is error handling. If the file cannot be 
transferred, or there is a typo, or disk error, HTCondor will notice and 
your job won't start. Should such an error happen with a shared 
filesystem, it probably won't happen until after your job starts, and it 
becomes the job's responsibility not just to detect the error, but to 
properly propagate the error up and out to HTCondor, so it can re-run 
the job. This is often hard to do, especially if you use 3rd party 
software. Usually what ends up happening is that the error is not 
correctly propagated out, and the job leaves the queue without correct 
or complete output, leading to very hard to debug problems. (Or worse, 
quietly missing data)

If you are using the native file transfer mechanism (i.e. not an URL), 
then file transfer is throttled by the access point. If using shared 
network filesystems, it is often possible for a lot of concurrent access 
to crash the file server or otherwise cause several performance problems.

HTCondor records in the job ad the number of input and output bytes 
transferred, which can be useful in determining how to size and 
provision network and disk size and bandwidth. This is harder to 
measure if using a shared file system.

Now, there's no such thing a free lunch. It is often difficult to know 
a-priori what the input file set it, in which case a shared filesystem 
might make more sense. Also, in the case where a job just needs a very 
small subset of a very large file, there may be performance benefits to 
reading that small chunk from a share filesystem instead of asking 
HTCondor to copy the whole file over, just to access a small part of it.


-greg