Re: [HTCondor-users] Caching large executable on worker nodes

[Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>]

On 08/12/2015 11:57 AM, Krieger, Donald N. wrote:

> If it doesn't exist, then fetch it.

If it doesn't exist,

1. look for a lock file.
2. If it exists, read the PID from the lock file and check if that exists.
3. If it does, sleep for a while and goto 1.
4. If it doesn't, create the lock file, start the fetch, write the pid
of your curl/wget to the lock file.
5. Wait for transfer to finish.

6. If the file does exist, it doesn't mean the transfer's finished or
was complete/successful. So you want to checksum it and so on.

So yes, it's pretty simple until it breaks. And then it gets complicated
fast.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Attachment: signature.asc
Description: OpenPGP digital signature