[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] Using Dropbox has a Condor pool job submission mechanism
- Date: Mon, 20 Jul 2009 09:34:32 +0100
- From: Ian Cottam <ian.cottam@xxxxxxxxxxxxxxxx>
- Subject: Re: [Condor-users] Using Dropbox has a Condor pool job submission mechanism
Thanks very much for that Ian.
I don't know Andy in Physics but I do know many colleagues in Research
Computing Services here in Manchester. In fact, I just sent them the
'thought variation' below for comment:
I've had a thought about how we could make it simpler for HTC/HPC
users to submit jobs to the local cloud of resources we have now and
that will doubtless grow considerably.
It involves using Dropbox <www.getdropbox.com>. I expect you are
familiar with it, but, if not, it is a great cross-platform file sync
(+backup) system. Anyone can get a 2GB account free and you can rent
50/100GB accounts for a small amount (I have a 50GB one for my 4 Macs
My 'thought experiment' really needs a diagram, but I'm useless with
such (especially within emails!).
Each Condor Pool, Grid, or HPC is usually front-ended with a node that
is also on the global network (so people can ssh in, or similar). I
propose that these nodes have paid 50/100GB Dropbox accounts. When a
user requests access to a facility, the relevant node shares a free 2GB
folder with him/her. (One user could have several such shares across the
range of facilities on campus.)
When the user want to run, say, a Condor job he/she simply moves or
copies the set of files necessary to the shared folder. Dropbox would
then automatically kick in and make sure it was synchronised with the
corresponding folder on the head node. All we would have to write is a
simple script that looks for such jobs arriving and submit them to the
When the job finishes and because the output files go to the shared
folder they will be copied by DropBox to the user's PC.
1. Users need not install systems, such as Condor, to use them.
2. The interface is trivial and uniform across all the facilities (job
submission files may differ of course).
3. There is no need to have user accounts on the resources (YES!).
4. There is no need to login/ssh/sftp/etc to the resource (which may be
running a platform foreign to the user).
5. If the job fails, the user only gets the log back; there is no
interaction with the job.
6. The script to monitor and then submit the job needs to know all the
files have arrived. This just needs some agreed protocol, but no doubt
there would be (other) subtleties of implementation.
Ian Stokes-Rees wrote:
I have spent quite a bit of time thinking of something very similar to
this. You are using the file system as the message queue. It is quite
a reasonable approach, in my opinion.
I'm not sure it is necessary to do anything which is "dropbox" specific
-- dropbox may just add on additional features (e.g. auto-syncing shared
files back to user machines).
You could easily imagine clients that are simply configured with account
access details and a path, and all the rest is based on configuration
files and job files found in particular directories.
If you have someone at Manchester who is interested in putting together
a proof of concept around this, I'd be very happy to contribute.
Do you know Andrew McNab in the Physics department? He is, in my
opinion, quite a visionary thinker when it comes to HPC/grid computing,
and has been "in the game" for a long time. If you can get a bit of his
time, he may be able to give you some good ideas/comments.
Ian Cottam wrote:
Has anyone thought about, or tried, using Dropbox <www.getdropbox.com>
as a job submission mechanism to a Condor pool?
Ian Stokes-Rees W: http://sbgrid.org
ijstokes@xxxxxxxxxxxxxxxxxxx T: +1 617 432-5608 x75
SBGrid, Harvard Medical School F: +1 617 432-5600
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
You can also unsubscribe by visiting
The archives can be found at:
Information Systems Manager
Manchester Interdisciplinary Biocentre &
School of Materials &
School of Chemical Engineering and Analytical Science (Room C24)
The John Garside Building (Room G.002)
The University of Manchester
t: 0161 306 5198
m: 07856 849831
[Giving a talk in MIB? Upload it: http://ic.mib.man.ac.uk/mibtheatre.php]