Re: [Condor-users] Python and Condor

[Nick LeRoy <nleroy@xxxxxxxxxxx>]

On Wed March 28 2007, simon kagwe wrote:
> Hi,
Hello,

>   I am supposed to write a program to run on a Condor grid on x86 Linux
> (Fedora Core 1 and Ubuntu) platform. The task is to show change in
> computation time when running a compute-intensive program on a grid as
> opposed to a single computer. Python is my programming language of choice,
> but there are a few things I need to know before I can start off:
>
>   1)Is it possible to run a Python program in standard universe? ( i want
> to take advantage of checkpointing and remote system calls)

Well, yes and no...  There is no native support for Python in Condor, and 
there is no way to run a python program (indeed, any interpreted program) in 
the standard universe.

That being said, if you're willing to do some extra work, you can still do it.

The first option is to use Python's built in pickle to checkpoint, and to use 
chirp for remote I/O.

Option #2:  Build your own python interpreter as a Condor standard universe 
executable, then submit your job with "executable = mypython" and "arguments 
= myscript.py" ...

> 2)Will I need 
> to use any parallel programming techniques like message-passing or BSP for
> the python program to run on the grid in such a way as to reduce
> computation time?

Depends on your program more than anything.  Does your program naturally split 
up so that multiple instances run independently of each other?  Look at 
section 2.4 of the Condor manual "Road-map for Running Jobs" 
http://www.cs.wisc.edu/condor/manual/v6.8.4/2_4Road_map_Running.html for more 
advice.

> 3)Will there be need to relink seeing as Python is an 
> interpreted language?

See above.  You can't relink your python program, but you'd have to relink 
your python interpreter itself.

> 4)Are there interfaces/bindings that I will need to 
> enable a python script to run on the grid (like Pyglobus for Globus
> toolkit).

I don't think that you'd need any of these.

>   If you can help me with the above questions or if you have any useful
> information about running python scripts on a Condor grid, please share.

Hope this helps

-Nick

-- 
           <<< The Matrix is everywhere. >>>
 /`-_    Nicholas R. LeRoy               The Condor Project
{     }/ http://www.cs.wisc.edu/~nleroy  http://www.cs.wisc.edu/condor
 \    /  nleroy@xxxxxxxxxxx              The University of Wisconsin
 |_*_|   608-265-5761                    Department of Computer Sciences