RE: [Condor-users] Forcing condor to re-use directories?

["Matt Hope" <Matt.Hope@xxxxxxxx>]

open/close a port instead?

further hackery :¬)

note that if you are running SMP then this may halve throughput if you lock for the entire length of the job...

> -----Original Message-----
> From: condor-users-bounces@xxxxxxxxxxx
> [mailto:condor-users-bounces@xxxxxxxxxxx]On Behalf Of Ian Chesal
> Sent: 30 June 2004 16:19
> To: Condor-Users Mail List
> Subject: RE: [Condor-users] Forcing condor to re-use directories?
> 
> 
> This comes very close to the Perl scripted solution I am using now,
> where I have the script move out of the condor execution directory and
> flock a temporary directory to run the job in. When the job is done it
> removes the file lock so another job can reuse the directory.
> 
> I was hoping Condor had some sort of force-reuse flag though. It would
> make my life easier as flock doesn't always work 100% perfectly on
> Windows.
> 
> Ian
> 
> > -----Original Message-----
> > From: condor-users-bounces@xxxxxxxxxxx 
> > [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Matt Hope
> > Sent: June 30, 2004 11:06 AM
> > To: Condor-Users Mail List
> > Subject: RE: [Condor-users] Forcing condor to re-use directories?
> > 
> > 
> > Much simpler (though hacky solution)
> > 
> > Make a networked storage area available to the condor users 
> > (this may involve further hackery with net use in bat files 
> > depending how secure you wish to be) and place the files there
> > 
> > Either work directly over the network (if it's fast enough for you)
> > 
> > or
> > 
> > Make a temp area available on every machine you will run on 
> > which is writable to any local user Again in a bat file check 
> > if the data is already copied (beware - ensure that the copy 
> > fully completed - remember that the process may have been 
> > pre-empted / killed / running on another vm at the same time, 
> > in which case you will need to lock on a local file) if not 
> > perform the copy. Call whatever exe you wanted to.
> > 
> > Note that this approach normally requires using an alternate 
> > batch file as the exe and transferring the script / exe you 
> > were going run as an additional file.
> > 
> > Unless your farm is homogenous and controlled by yourself 
> > entirely you will want to add an additional flag to the 
> > machine class ads to indicate they have the requisite space 
> > and access to allow this and add it to the requirements of 
> your jobs.
> > 
> > As I said this is very hacky and certainly carries security 
> > concerns but will get it working very fast.
> > 
> > A facility on condor to locally cache specified files as 
> > needed / able based on avail disk space would be nice
> > 
> > Matt
> > 
> > 
> > > -----Original Message-----
> > > From: condor-users-bounces@xxxxxxxxxxx 
> > > [mailto:condor-users-bounces@xxxxxxxxxxx]On Behalf Of Mark 
> > Silberstein
> > > Sent: 30 June 2004 15:54
> > > To: Condor-Users Mail List; ICHESAL@xxxxxxxxxx
> > > Cc: Gabi Kliot
> > > Subject: Re: [Condor-users] Forcing condor to re-use directories?
> > > 
> > > 
> > > Ian,
> > > You might be interested in add-on which does it for you 
> > automatically. 
> > > We wrote such add-on for Condor ( only for Linux now, but we can 
> > > consider adding Windows Support if you want )  which caches 
> > > executables and does I/O pipelining.
> > > Take a look at 
> > > http://www.cs.technion.ac.il/Labs/dsl/completed_projects/condo
> > > r-llic/llic_web_site.htm
> > > 
> > > Reply directly to me if you have any further questions
> > > Mark
> > > 
> > > On Mon, 2004-06-28 at 20:46, Ian Chesal wrote:
> > > > Is there a way to force condor to reuse a particular
> > > directory for each
> > > > virtual machine instance on a client when it executes 
> > things for you 
> > > > instead of creating a new temp directory? We have 
> > experiments that 
> > > > transfer and operate on 1GB+ worth of data. Since most
> > > experiments share
> > > > the same data we'd like to just re-use existing data. I
> > > could write some
> > > > directory sharing symantecs into my control script that
> > > runs each job,
> > > > but I thought I'd find out if Condor supports this 
> > concept natively 
> > > > first.
> > > > 
> > > > Cheers!
> > > > Ian
> > > > 
> > > > --
> > > > Ian R. Chesal <ichesal@xxxxxxxxxx>
> > > > Advanced Software Engineer
> > > > 
> > > > Altera Corporation
> > > > Toronto Technology Center
> > > > Tel: (416) 926-8300
> > > > 
> > > > _______________________________________________
> > > > Condor-users mailing list
> > > > Condor-users@xxxxxxxxxxx 
> > > > http://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > > 
> > > _______________________________________________
> > > Condor-users mailing list
> > > Condor-users@xxxxxxxxxxx 
> > > http://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > > 
> > 
> > 
> > *****************************************************************
> > Gloucester Research Limited believes the information 
> > provided herein is reliable. While every care has been 
> > taken to ensure accuracy, the information is furnished 
> > to the recipients with no warranty as to the completeness 
> > and accuracy of its contents and on condition that any 
> > errors or omissions shall not be made the basis for any 
> > claim, demand or cause for action.
> > *****************************************************************
> > 
> > _______________________________________________
> > Condor-users mailing list
> > Condor-users@xxxxxxxxxxx 
> > http://lists.cs.wisc.edu/mailman/listinfo/cond> or-users
> > 
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx
> http://lists.cs.wisc.edu/mailman/listinfo/condor-users
>