[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Using Dropbox has a Condor pool job submission mechanism

Thanks very much for that Ian.
I don't know Andy in Physics but I do know many colleagues in Research Computing Services here in Manchester. In fact, I just sent them the 'thought variation' below for comment:
Hi ...,
I've had a thought about how we could make it simpler for HTC/HPC users to submit jobs to the local cloud of resources we have now and that will doubtless grow considerably.

It involves using Dropbox <www.getdropbox.com>. I expect you are familiar with it, but, if not, it is a great cross-platform file sync (+backup) system. Anyone can get a 2GB account free and you can rent 50/100GB accounts for a small amount (I have a 50GB one for my 4 Macs for example).

My 'thought experiment' really needs a diagram, but I'm useless with such (especially within emails!).

Each Condor Pool, Grid, or HPC is usually front-ended with a node that is also on the global network (so people can ssh in, or similar). I propose that these nodes have paid 50/100GB Dropbox accounts. When a user requests access to a facility, the relevant node shares a free 2GB folder with him/her. (One user could have several such shares across the range of facilities on campus.)

When the user want to run, say, a Condor job he/she simply moves or copies the set of files necessary to the shared folder. Dropbox would then automatically kick in and make sure it was synchronised with the corresponding folder on the head node. All we would have to write is a simple script that looks for such jobs arriving and submit them to the job queue.

When the job finishes and because the output files go to the shared folder they will be copied by DropBox to the user's PC.

Some consequences:
1. Users need not install systems, such as Condor, to use them.
2. The interface is trivial and uniform across all the facilities (job submission files may differ of course).
3. There is no need to have user accounts on the resources (YES!).
4. There is no need to login/ssh/sftp/etc to the resource (which may be running a platform foreign to the user). 5. If the job fails, the user only gets the log back; there is no interaction with the job. 6. The script to monitor and then submit the job needs to know all the files have arrived. This just needs some agreed protocol, but no doubt there would be (other) subtleties of implementation.


Ian Stokes-Rees wrote:
Hi Ian,

I have spent quite a bit of time thinking of something very similar to this. You are using the file system as the message queue. It is quite a reasonable approach, in my opinion.

I'm not sure it is necessary to do anything which is "dropbox" specific -- dropbox may just add on additional features (e.g. auto-syncing shared files back to user machines).

You could easily imagine clients that are simply configured with account access details and a path, and all the rest is based on configuration files and job files found in particular directories.

If you have someone at Manchester who is interested in putting together a proof of concept around this, I'd be very happy to contribute.

Do you know Andrew McNab in the Physics department? He is, in my opinion, quite a visionary thinker when it comes to HPC/grid computing, and has been "in the game" for a long time. If you can get a bit of his time, he may be able to give you some good ideas/comments.


Ian Cottam wrote:
Has anyone thought about, or tried, using Dropbox <www.getdropbox.com> as a job submission mechanism to a Condor pool?

Ian Stokes-Rees                            W: http://sbgrid.org
ijstokes@xxxxxxxxxxxxxxxxxxx               T: +1 617 432-5608 x75
SBGrid, Harvard Medical School             F: +1 617 432-5600


Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: https://lists.cs.wisc.edu/archive/condor-users/

Ian Cottam
Information Systems Manager
Manchester Interdisciplinary Biocentre &
School of Materials &
School of Chemical Engineering and Analytical Science (Room C24)
The John Garside Building (Room G.002)
The University of Manchester
e: ian.cottam@xxxxxxxxxxxxxxxx
t: 0161 306 5198
m: 07856 849831
[Giving a talk in MIB? Upload it: http://ic.mib.man.ac.uk/mibtheatre.php]