[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [Condor-users] Forcing condor to re-use directories?

Much simpler (though hacky solution)

Make a networked storage area available to the condor users (this may involve further hackery with net use in bat files depending how secure you wish to be) and place the files there

Either work directly over the network (if it's fast enough for you)


Make a temp area available on every machine you will run on which is writable to any local user
Again in a bat file check if the data is already copied (beware - ensure that the copy fully completed - remember that the process may have been pre-empted / killed / running on another vm at the same time, in which case you will need to lock on a local file)
if not perform the copy.
Call whatever exe you wanted to.

Note that this approach normally requires using an alternate batch file as the exe and transferring the script / exe you were going run as an additional file.

Unless your farm is homogenous and controlled by yourself entirely you will want to add an additional flag to the machine class ads to indicate they have the requisite space and access to allow this and add it to the requirements of your jobs.

As I said this is very hacky and certainly carries security concerns but will get it working very fast.

A facility on condor to locally cache specified files as needed / able based on avail disk space would be nice


> -----Original Message-----
> From: condor-users-bounces@xxxxxxxxxxx
> [mailto:condor-users-bounces@xxxxxxxxxxx]On Behalf Of Mark Silberstein
> Sent: 30 June 2004 15:54
> To: Condor-Users Mail List; ICHESAL@xxxxxxxxxx
> Cc: Gabi Kliot
> Subject: Re: [Condor-users] Forcing condor to re-use directories?
> Ian,
> You might be interested in add-on which does it for you automatically.
> We wrote such add-on for Condor ( only for Linux now, but we can
> consider adding Windows Support if you want )  which caches 
> executables
> and does I/O pipelining.
> Take a look at 
> http://www.cs.technion.ac.il/Labs/dsl/completed_projects/condo
> r-llic/llic_web_site.htm
> Reply directly to me if you have any further questions
> Mark
> On Mon, 2004-06-28 at 20:46, Ian Chesal wrote:
> > Is there a way to force condor to reuse a particular 
> directory for each
> > virtual machine instance on a client when it executes things for you
> > instead of creating a new temp directory? We have experiments that
> > transfer and operate on 1GB+ worth of data. Since most 
> experiments share
> > the same data we'd like to just re-use existing data. I 
> could write some
> > directory sharing symantecs into my control script that 
> runs each job,
> > but I thought I'd find out if Condor supports this concept natively
> > first.
> > 
> > Cheers!
> > Ian
> > 
> > --
> > Ian R. Chesal <ichesal@xxxxxxxxxx>
> > Advanced Software Engineer
> > 
> > Altera Corporation
> > Toronto Technology Center
> > Tel: (416) 926-8300
> > 
> > _______________________________________________
> > Condor-users mailing list
> > Condor-users@xxxxxxxxxxx
> > http://lists.cs.wisc.edu/mailman/listinfo/condor-users
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx
> http://lists.cs.wisc.edu/mailman/listinfo/condor-users

Gloucester Research Limited believes the information 
provided herein is reliable. While every care has been 
taken to ensure accuracy, the information is furnished 
to the recipients with no warranty as to the completeness 
and accuracy of its contents and on condition that any 
errors or omissions shall not be made the basis for any 
claim, demand or cause for action.