[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Caching large executable on worker nodes

 here at Manchester we wrote this for large data files but you could use
the same code for caching and accessing executables.
On 11/08/2015 21:38, "HTCondor-users on behalf of Jens Schmaler"
<htcondor-users-bounces@xxxxxxxxxxx on behalf of jens.schmaler@xxxxxx>

>Hi all,
>we are currently in a situation where transferring the executable to the
>execute machine for each job starts to get a limiting factor. Our case
>is the following:
>- large executable (500MB), which is the same for a large number of jobs
>within one cluster (jobs only differ in input arguments)
>- few execute machines, i.e. each execute machine will run many such
>jobs (transferring the executable each time although this would not be
>- we are using the file transfer mechanism, but I believe the problem
>would be similar with a shared file system
>- we would like to keep the current job structure for various reasons,
>i.e. we would rather not combine multiple jobs into one longer-running
>one (I can provide the arguments for this if needed)
>My goal would be to reduce the time and network traffic for transferring
>the executable thousands of times.
>A very natural idea would be to cache the executable on each execute
>machine, hoping that we can make use of it in case we get another job of
>the same cluster. I probably would be able to hack something that will
>do the trick, although doing it properly might take quite some effort
>(when and how to clean up the cache?, ...)
>On the other hand, this seems like a very common problem, so I was
>wondering whether Condor offers some built-in magic to cope with this?
>Maybe I am missing something obvious?
>Are there any recommended best practices for my case?
>Thank you very much in advance,
>HTCondor-users mailing list
>To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with
>subject: Unsubscribe
>You can also unsubscribe by visiting
>The archives can be found at:

Ian Cottam  | IT Relationship Manager | IT Services  | C38 Sackville
Street Building  |  The University of Manchester  |  M13 9PL  |
+44(0)161 306 1851