[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [condor-users] globus universe question



The problem Mark is describing was caused by the fact that the Condor pool
he was submitting to (via the globus universe) didn't have a shared
filesystem, a situation that Globus doesn't handle well. If you're not in
that situation, you shouldn't have to jump through the hoops he's had to.

-- Jaime

On Tue, 9 Dec 2003, Mark Calleja wrote:

> Hi Sebastian,
>
> I also came up against the problem of retrieving output files from
> condor-g jobs (see a previous thread in last month's mailing list), and
> the only way I got round it was to amend the condor.pm file in globus so
> that a new process is forked by the jobmanager which is then exec'd with
> a monitoring process. This process waits for the condor job to finish
> before using gsiftp to return all output files back to the submitting
> machine. Not pretty, but it works. If you're interested in this route
> then drop me a line and I'll give you what I've done.
>
> Cheers,
>
> Mark Calleja
> --
> Department of Earth Sciences, University of Cambridge
> Downing Street, Cambridge CB2 3EQ, UK
> Tel. (+44/0) 1223 333408, Fax  (+44/0) 1223 333450
> http://www.esc.cam.ac.uk/~mcal00
>
> On Mon, 2003-12-08 at 20:25, Jaime Frey wrote:
> > On Mon, 8 Dec 2003, Sebastian Grinstein wrote:
> >
> > > Hello Condor users and experts,
> > >
> > > I'm starting to use condor. I have simple question:
> > >
> > > I submit a job in the globus universe, the executable generates
> > > an output file. How do I retrieve this file (or files)?
> > >
> > > This is the submission file I'm using:
> > > > cat tio_globus.sdf
> > > # Condor submit description file
> > > Executable           = tio
> > > Universe             = globus
> > > globusscheduler      = remote.machine.edu
> > > transfer_input       = true
> > > transfer_input_files = arg_tio.i
> > > transfer_files       = ONEXIT
> > > transfer_output      = true
> > > Input = arg_tio.i
> > > Arguments = arg_tio.i arg_gtio.o
> > > Output = /tmp/gtio.out
> > > Error  = /tmp/gtio.err
> > > Log    = /tmp/gtio.log
> > > Queue
> > >
> > > "tio in out" copies the contents of the first file (in) into the
> > > second (out).
> > >
> > > When I condor_submit, in the remote.machine, I see the job being done
> > > in .globus/.gass_cache/...
> > >
> > > However, when the job is done the .globus/.gass_cache/ files are
> > > removed, and condor gives me the gtio.* files, but how do I get
> > > arg_gtio.o ?
> > >
> > > Of course there were no errors reported. If I try a
> > > "globus-job-get-output" (with the id from the logs) I get a "Invalid
> > > job id." (while "globus-job-status" says "DONE").
> >
> > You need to specify "arg_gtio.o" with the transfer_output_files parameter
> > in your submit description file. In most universes in Condor, if you omit
> > transfer_output_files, Condor automatically transfers back all
> > created/modified files in the job's execute directory. This doesn't work
> > in the globus universe, as Globus doesn't an appropriate mechanism.
> >
> > I see that this exception isn't well documented in the Condor manual. I'll
> > see that we fix it.
> >
> > +------------------------------------+-------------------------------+
> > |             Jaime Frey             |There are 10 types of people in|
> > |         jfrey@xxxxxxxxxxx          |the world: Those who understand|
> > |   http://www.cs.wisc.edu/~jfrey/   |  binary, and those who don't  |
> > +------------------------------------+-------------------------------+
> > Condor Support Information:
> > http://www.cs.wisc.edu/condor/condor-support/
> > To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
> > unsubscribe condor-users <your_email_address>
>
>
>
> Condor Support Information:
> http://www.cs.wisc.edu/condor/condor-support/
> To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
> unsubscribe condor-users <your_email_address>
>

+------------------------------------+-------------------------------+
|             Jaime Frey             |There are 10 types of people in|
|         jfrey@xxxxxxxxxxx          |the world: Those who understand|
|   http://www.cs.wisc.edu/~jfrey/   |  binary, and those who don't  |
+------------------------------------+-------------------------------+
Condor Support Information:
http://www.cs.wisc.edu/condor/condor-support/
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>