RE: [Condor-users] Forcing condor to re-use directories?

["Ian Chesal" <ICHESAL@xxxxxxxxxx>]

This comes very close to the Perl scripted solution I am using now,
where I have the script move out of the condor execution directory and
flock a temporary directory to run the job in. When the job is done it
removes the file lock so another job can reuse the directory.

I was hoping Condor had some sort of force-reuse flag though. It would
make my life easier as flock doesn't always work 100% perfectly on
Windows.

Ian

> -----Original Message-----
> From: condor-users-bounces@xxxxxxxxxxx 
> [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Matt Hope
> Sent: June 30, 2004 11:06 AM
> To: Condor-Users Mail List
> Subject: RE: [Condor-users] Forcing condor to re-use directories?
> 
> 
> Much simpler (though hacky solution)
> 
> Make a networked storage area available to the condor users 
> (this may involve further hackery with net use in bat files 
> depending how secure you wish to be) and place the files there
> 
> Either work directly over the network (if it's fast enough for you)
> 
> or
> 
> Make a temp area available on every machine you will run on 
> which is writable to any local user Again in a bat file check 
> if the data is already copied (beware - ensure that the copy 
> fully completed - remember that the process may have been 
> pre-empted / killed / running on another vm at the same time, 
> in which case you will need to lock on a local file) if not 
> perform the copy. Call whatever exe you wanted to.
> 
> Note that this approach normally requires using an alternate 
> batch file as the exe and transferring the script / exe you 
> were going run as an additional file.
> 
> Unless your farm is homogenous and controlled by yourself 
> entirely you will want to add an additional flag to the 
> machine class ads to indicate they have the requisite space 
> and access to allow this and add it to the requirements of your jobs.
> 
> As I said this is very hacky and certainly carries security 
> concerns but will get it working very fast.
> 
> A facility on condor to locally cache specified files as 
> needed / able based on avail disk space would be nice
> 
> Matt
> 
> 
> > -----Original Message-----
> > From: condor-users-bounces@xxxxxxxxxxx 
> > [mailto:condor-users-bounces@xxxxxxxxxxx]On Behalf Of Mark 
> Silberstein
> > Sent: 30 June 2004 15:54
> > To: Condor-Users Mail List; ICHESAL@xxxxxxxxxx
> > Cc: Gabi Kliot
> > Subject: Re: [Condor-users] Forcing condor to re-use directories?
> > 
> > 
> > Ian,
> > You might be interested in add-on which does it for you 
> automatically. 
> > We wrote such add-on for Condor ( only for Linux now, but we can 
> > consider adding Windows Support if you want )  which caches 
> > executables and does I/O pipelining.
> > Take a look at 
> > http://www.cs.technion.ac.il/Labs/dsl/completed_projects/condo
> > r-llic/llic_web_site.htm
> > 
> > Reply directly to me if you have any further questions
> > Mark
> > 
> > On Mon, 2004-06-28 at 20:46, Ian Chesal wrote:
> > > Is there a way to force condor to reuse a particular
> > directory for each
> > > virtual machine instance on a client when it executes 
> things for you 
> > > instead of creating a new temp directory? We have 
> experiments that 
> > > transfer and operate on 1GB+ worth of data. Since most
> > experiments share
> > > the same data we'd like to just re-use existing data. I
> > could write some
> > > directory sharing symantecs into my control script that
> > runs each job,
> > > but I thought I'd find out if Condor supports this 
> concept natively 
> > > first.
> > > 
> > > Cheers!
> > > Ian
> > > 
> > > --
> > > Ian R. Chesal <ichesal@xxxxxxxxxx>
> > > Advanced Software Engineer
> > > 
> > > Altera Corporation
> > > Toronto Technology Center
> > > Tel: (416) 926-8300
> > > 
> > > _______________________________________________
> > > Condor-users mailing list
> > > Condor-users@xxxxxxxxxxx 
> > > http://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > 
> > _______________________________________________
> > Condor-users mailing list
> > Condor-users@xxxxxxxxxxx 
> > http://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > 
> 
> 
> *****************************************************************
> Gloucester Research Limited believes the information 
> provided herein is reliable. While every care has been 
> taken to ensure accuracy, the information is furnished 
> to the recipients with no warranty as to the completeness 
> and accuracy of its contents and on condition that any 
> errors or omissions shall not be made the basis for any 
> claim, demand or cause for action.
> *****************************************************************
> 
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx 
> http://lists.cs.wisc.edu/mailman/listinfo/cond> or-users
>