[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Condor-users] Using Dropbox has a Condor pool job submission mechanism

Has anyone thought about, or tried, using Dropbox <www.getdropbox.com> as a job submission mechanism to a Condor pool?

What I have in mind is the sort of pool that comprises some large number of nodes on a private network, with one node -- the submit node -- having two network cards: the second being on the global network (i.e. the visible Internet). Every user has an account on the submit node, and they have to ssh in (or similar) and copy files over; submit the job; monitor for it finishing; copy results out.

(If the submit node is also on a shared network file system, also mounted on users' machines, the below is invalidated, but I think there are many instances where that is not the case.)

The scenario is that all the users use 2GB free Dropbox accounts. (It would still work if, like me, they already had [paid] accounts). The single job submit node 'owns' a paid account of 50 or 100GB. When one becomes a user of the pool, a folder is shared, via Dropbox, from the job submit node account with the user's account. Users do not need login accounts on the submit node.

Then, to submit a Condor job into the private pool, a user would simply move or copy the set of files required into the shared folder. A simple monitor program on the job submit node would need to look for jobs arriving and condor_submit them. The results simply appear back on the user's remote machine courtesy of the DropBox sync mechanism.

Clearly, there would be some subtleties of implementation; for example: a Condor job submit file would have to have a standardised name; and one would need a mechanism for ensuring DropBox had synchronised all the files before the job was run; and no doubt others.

Just a thought, but I would be interested in reading people's comments.

Ian Cottam
Information Systems Manager
Manchester Interdisciplinary Biocentre &
School of Materials &
School of Chemical Engineering and Analytical Science (Room C24)
The John Garside Building (Room G.002)
The University of Manchester
e: ian.cottam@xxxxxxxxxxxxxxxx
t: 0161 306 5198
m: 07856 849831
[Giving a talk in MIB? Upload it: http://ic.mib.man.ac.uk/mibtheatre.php]