Re: [HTCondor-users] Caching large executable on worker nodes

[Michael V Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

At HTCondor Week this year there were three
different presentations about input transfer caching including Carl's presentation
regarding Squid.

The first was "Efficently Sharing Common Data" by Zach Miller of CHTC, and the second was "Data Caching with HTCondor: The HTCondor CacheD" by Derek Weitzel with Brian Bockelman at the University of Nebraska - Lincoln, with Carl's rounding out the list.

I liked the CacheD approach because the CacheD is designed as just another part of the HTCondor ecosystem, meaning you can do things like schedule or evict it, set requirements, etc. In addition, he has worked on using a torrent-based transfer, where an exec node can, for example, fetch its inputs from one or more other peers which are already running the job.

http://research.cs.wisc.edu/htcondor/HTCondorWeek2015/presentations/WeitzelD_CacheDPres.pdf

The Wednesday presentations on "Data Management" started after lunch:

http://research.cs.wisc.edu/htcondor/HTCondorWeek2015/wednesday.html

Unfortunately my PowerPoint 2010 won't open Zach's presentation.

        -Michael Pelletier.