[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Python and Condor

Thank you Nick for your answers.
So does it mean the machines will have to share a filesystem.? I was trying to avoid using NFS or AFS. The whole reason I was going to implement it on Linux was to take advantage of checkpointing and remote system calls.

Nick LeRoy <nleroy@xxxxxxxxxxx> wrote:
On Wed March 28 2007, simon kagwe wrote:
> Hi,

> I am supposed to write a program to run on a Condor grid on x86 Linux
> (Fedora Core 1 and Ubuntu) platform. The task is to show change in
> computation time when running a compute-intensive program on a grid as
> opposed to a single computer. Python is my programming language of choice,
> but there are a few things I need to know before I can start off:
> 1)Is it possible to run a Python program in standard universe? ( i want
> to take advantage of checkpointing and remote system calls)

Well, yes and no... There is no native support for Python in Condor, and
there is no way to run a python program (indeed, any interpreted program) in
the standard universe.

That being said, if you're willing to do some extra work, you can still do it.

The first option is to use Python's built in pickle to checkpoint, and to use
chirp for remote I/O.

Option #2: Build your own python interpreter as a Condor standard universe
executable, then submit your job with "executable = mypython" and "arguments
= myscript.py" ...

> 2)Will I need
> to use any parallel programming techniques like message-passing or BSP for
> the python program to run on the grid in such a way as to reduce
> computation time?

Depends on your program more than anything. Does your program naturally split
up so that multiple instances run independently of each other? Look at
section 2.4 of the Condor manual "Road-map for Running Jobs"
http://www.cs.wisc.edu/condor/manual/v6.8.4/2_4Road_map_Running.html for more

> 3)Will there be need to relink seeing as Python is an
> interpreted language?

See above. You can't relink your python program, but you'd have to relink
your python interpreter itself.

> 4)Are there interfaces/bindings that I will need to
> enable a python script to run on the grid (like Pyglobus for Globus
> toolkit).

I don't think that you'd need any of these.

> If you can help me with the above questions or if you have any useful
> information about running python scripts on a Condor grid, please share.

Hope this helps


<<< The Matrix is everywhere. >>>
/`-_ Nicholas R. LeRoy The Condor Project
{ }/ http://www.cs.wisc.edu/~nleroy http://www.cs.wisc.edu/condor
\ / nleroy@xxxxxxxxxxx The University of Wisconsin
|_*_| 608-265-5761 Department of Computer Sciences
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at either

Be a PS3 game guru.
Get your game face on with the latest PS3 news and previews at Yahoo! Games.