When working on a single homogeneous compute cluster, are there any advantages to using HTCondor's file-transfer rather than working off shared network storage? I guess it would depend on the network and storage speeds.
It's just interesting that the "always work in local scratch" mindset I am used to is seen a serious backward step performance wise:
Scratch therefore only useful if your network storage or interconnects are slow or saturated ... copying bulk data to local storage / getting all users to copy to local/scratch storage is a quick way to saturate your storage infrastructure.I can find other instances of this HPC conventional wisdom and it intuitively makes sense. But I don't understand networked storage well, so I am asking the HTCondor hivemind for their thoughts.
-- Matthew T. West DevOps & HPC SysAdmin University of Exeter, Research IT www.exeter.ac.uk/research/researchcomputing/support/researchit 57 Laver Building, North Park Road, Exeter, EX4 4QE, United Kingdom Please note, I may send emails out of 'normal' working hours, as this fits my own work-life balance. I do not expect a response outside of your own working hours.