Re: [HTCondor-users] Caching large executable on worker nodes

[Ian Cottam <Ian.Cottam@xxxxxxxxxxxxxxxx>]

On 11/08/2015 22:16, "HTCondor-users on behalf of Jens Schmaler"
<htcondor-users-bounces@xxxxxxxxxxx on behalf of jens.schmaler@xxxxxx>
wrote:

>Hi Ian,
>
>yes, something like this is exactly what I would need to implement. I
>have no permission to access this script, but I would be very interested
>to learn how you handle the subtleties (concurrent downloads to the same
>machine, aborted downloads, clean-up etc.).
>
>Would you be willing to share some of your concepts? I am curious to see
>whether I can implement this on Windows, too.
>
>Thanks a lot, this is highly appreciated!
>
>Jens
>_______________________________________________
>HTCondor-users mailing list
>To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with
>a
>subject: Unsubscribe
>You can also unsubscribe by visiting
>https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
>The archives can be found at:
>https://lists.cs.wisc.edu/archive/htcondor-users/
>

We are Linux based. Here is the script we call: mirror
--
#!/bin/bash

filename=$2
username=$1

download="wget -nH -N -q --cut-dirs=1
http://submitter.eps.manchester.ac.uk/~$username/$filename";
lockfile=$filename".lockfile"
create_lockfile="touch $lockfile"
remove_lockfile="rm -f $lockfile"

cd /tmp
if [ ! -r $filename ]; then
   $create_lockfile ; $download ; $remove_lockfile
#  echo "Initial download completed"
   else
   if [ -f $lockfile ]; then
      if [ `find $lockfile -cmin +30` ]; then
          $create_lockfile ; $download ; $remove_lockfile
#        echo "Interrupted download completed"
          else
          while ( [ -r $lockfile ] ); do
                sleep 5
                if [ `find $filename -cmin +2` ]; then
                   $create_lockfile ; $download ; $remove_lockfile
#                  echo "Interrupted download completed"
                   exit 0
                fi
          done
#         echo "Ongoing download completed"
      fi
      else
      if [ `find $filename -cmin +5` ]; then
         $create_lockfile ; $download ; $remove_lockfile
#        echo "File download refreshed"
#        else
#        echo "Previous download reused"
      fi
   fi
fi



--
-- 
Ian Cottam  | IT Relationship Manager | IT Services  | C38 Sackville
Street Building  |  The University of Manchester  |  M13 9PL  |