Re: [HTCondor-users] Caching large executable on worker nodes

[Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>]

On 08/12/2015 11:11 AM, Jens Schmaler wrote:

> Still, I must admit that I do not fully understand the concept yet. Even
> with a SQUID cache for my cluster, my large executable will still be
> transferred over the network to the execute machine for each job. The
> SQUID server might take the load from the submit machine and ideally
> would have a better network bandwidth, but the overall network traffic
> remains.

If you're running the default 1 slot per core setup and have, say, 8
jobs running on the same node, you end up with 8 concurrent transfers of
the same file to the same machine. That'll choke your node's NIC and
potentially the switch's backplane (not with 500MB files of course) long
before that gets to the proxy server.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Attachment: signature.asc
Description: OpenPGP digital signature