[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] HTCondor file-transfer vs networked storage
- Date: Mon, 22 Aug 2022 14:37:14 -0500
- From: Nick LeRoy <leroy.nick@xxxxxxxxx>
- Subject: Re: [HTCondor-users] HTCondor file-transfer vs networked storage
On Sat, Aug 20, 2022 at 8:44 AM Matthew T West via HTCondor-users
> Hi All,
> When working on a single homogeneous compute cluster, are there any advantages to using HTCondor's file-transfer rather than working off shared network storage? I guess it would depend on the network and storage speeds.
> It's just interesting that the "always work in local scratch" mindset I am used to is seen a serious backward step performance wise:
> Scratch therefore only useful if your network storage or interconnects are slow or saturated ... copying bulk data to local storage / getting all users to copy to local/scratch storage is a quick way to saturate your storage infrastructure.
> I can find other instances of this HPC conventional wisdom and it intuitively makes sense. But I don't understand networked storage well, so I am asking the HTCondor hivemind for their thoughts.
You need to remember that HTCondor can work in many different
environments, among these being WANs, campus-type structures, and
grids. For these types of scenarios, file transfer is preferable, if