[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Condor-users] Using Dropbox has a Condor pool job submission mechanism
- Date: Fri, 17 Jul 2009 13:40:16 +0100
- From: Ian Cottam <ian.cottam@xxxxxxxxxxxxxxxx>
- Subject: [Condor-users] Using Dropbox has a Condor pool job submission mechanism
Has anyone thought about, or tried, using Dropbox <www.getdropbox.com>
as a job submission mechanism to a Condor pool?
What I have in mind is the sort of pool that comprises some large number
of nodes on a private network, with one node -- the submit node --
having two network cards: the second being on the global network (i.e.
the visible Internet). Every user has an account on the submit node, and
they have to ssh in (or similar) and copy files over; submit the job;
monitor for it finishing; copy results out.
(If the submit node is also on a shared network file system, also
mounted on users' machines, the below is invalidated, but I think there
are many instances where that is not the case.)
The scenario is that all the users use 2GB free Dropbox accounts. (It
would still work if, like me, they already had [paid] accounts). The
single job submit node 'owns' a paid account of 50 or 100GB.
When one becomes a user of the pool, a folder is shared, via Dropbox,
from the job submit node account with the user's account. Users do not
need login accounts on the submit node.
Then, to submit a Condor job into the private pool, a user would simply
move or copy the set of files required into the shared folder. A simple
monitor program on the job submit node would need to look for jobs
arriving and condor_submit them. The results simply appear back on the
user's remote machine courtesy of the DropBox sync mechanism.
Clearly, there would be some subtleties of implementation; for example:
a Condor job submit file would have to have a standardised name; and one
would need a mechanism for ensuring DropBox had synchronised all the
files before the job was run; and no doubt others.
Just a thought, but I would be interested in reading people's comments.
Information Systems Manager
Manchester Interdisciplinary Biocentre &
School of Materials &
School of Chemical Engineering and Analytical Science (Room C24)
The John Garside Building (Room G.002)
The University of Manchester
t: 0161 306 5198
m: 07856 849831
[Giving a talk in MIB? Upload it: http://ic.mib.man.ac.uk/mibtheatre.php]