[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Caching large executable on worker nodes

On 08/11/2015 01:38 PM, Jens Schmaler wrote:

> On the other hand, this seems like a very common problem, so I was
> wondering whether Condor offers some built-in magic to cope with this?
> Maybe I am missing something obvious?

Nothing that I know of.

> Are there any recommended best practices for my case?

In our case the problem is 100GB (and counting) of data files that
change weekly. So far I tried scp, btsync/syncthing, lsyncd, ceph,
gluster: they all suck. Some day I'll try data transfers as parent dag
nodes on whole machine slots...

If you have one .5G executable that doesn't change, and the number of
execute nodes doesn't change too often, I'd just install it on every
node and have some version of HAS_MY_SOFTWARE (look for it in the
Subjects of recent list posts) in the classad.

Dimitri Maziuk
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Attachment: signature.asc
Description: OpenPGP digital signature