[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Caching large executable on worker nodes

At HTCondor Week this year there were three different presentations about input transfer caching including Carl's presentation regarding Squid.

The first was "Efficently Sharing Common Data" by Zach Miller of CHTC, and the second was "Data Caching with HTCondor: The HTCondor CacheD" by Derek Weitzel with Brian Bockelman at the University of Nebraska - Lincoln, with Carl's rounding out the list.

I liked the CacheD approach because the CacheD is designed as just another part of the HTCondor ecosystem, meaning you can do things like schedule or evict it, set requirements, etc. In addition, he has worked on using a torrent-based transfer, where an exec node can, for example, fetch its inputs from one or more other peers which are already running the job.


The Wednesday presentations on "Data Management" started after lunch:


Unfortunately my PowerPoint 2010 won't open Zach's presentation.

        -Michael Pelletier.