Re: [HTCondor-users] Caching large executable on worker nodes

[Derek Weitzel <dweitzel@xxxxxxxxxxx>]

On Aug 12, 2015, at 3:31 PM, Michael V Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx> wrote:
> 
> The CacheD was sardonically described as "grad-student-ware" by the author - quite similar to a feasibility study, but with a shiny cap and gown at the end of it if it's done right. 
> 
> It's certainly a common concern among the entire user community, so I think that the Squid capability is just the first step in this direction. It may well turn out that CacheD or something like it will mature enough to join the main branch at some point in the future. 

As the author of said CacheD, you are correct, it has a cap / gown at the end of it (I graduate on Saturday).  

In the future, HTCondor may have a CacheD-like ability (if not, exactly the CacheD).  The CacheD is a large undertaking, and therefore will need a lot of work before itâs ready for use in production.  But, it would be an elegant solution to this problem.  

In the mean time, if your goal is to reduce the number of times that an executable is transferred to a worker node, there isnât many easy-to-use options.  I donât know of anything built into HTCondor that can do node local caching. 

There are some big pieces of software that allow node local caching, such as CernVM-FS.  But, again, itâs a lot of infrastructure to deploy for this singular problem.

Sorry I donât have a great answer for you at this time.  But, in the future, there may be a great option for you!

-Derek

Attachment: smime.p7s
Description: S/MIME cryptographic signature