Re: [HTCondor-users] Caching large executable on worker nodes

[Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>]

On 08/11/2015 01:38 PM, Jens Schmaler wrote:

> On the other hand, this seems like a very common problem, so I was
> wondering whether Condor offers some built-in magic to cope with this?
> Maybe I am missing something obvious?

Nothing that I know of.

> Are there any recommended best practices for my case?

In our case the problem is 100GB (and counting) of data files that
change weekly. So far I tried scp, btsync/syncthing, lsyncd, ceph,
gluster: they all suck. Some day I'll try data transfers as parent dag
nodes on whole machine slots...

If you have one .5G executable that doesn't change, and the number of
execute nodes doesn't change too often, I'd just install it on every
node and have some version of HAS_MY_SOFTWARE (look for it in the
Subjects of recent list posts) in the classad.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Attachment: signature.asc
Description: OpenPGP digital signature