Re: [Condor-users] retrieving output subdirectories

["Smith, Ian" <I.C.Smith@xxxxxxxxxxxxxxx>]

You need to put the tarring in a shell script/batch file that will
run on the *execute host* not the submit host. If you do this
immediately after the application finishes the files will still be
there. You can get condor to just transfer the tar file using
transfer_output_files in your .sub file.

-ian.




> -----Original Message-----
> From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-
> bounces@xxxxxxxxxxx] On Behalf Of Javier Forment
> Sent: 04 February 2008 16:16
> To: Condor-Users Mail List
> Subject: Re: [Condor-users] retrieving output subdirectories
> 
> Excuse my ignorance, but... I can't see how to do that: if I run my
> application with condor_submit, Condor run it, and, when finished, it
> moves back all the output files in the plain execute directory (ignoring
> subdirectories), and remove all the files in the execute directory
> (including subdirectories). If I put the tarring of subdirectory output
> files just after condor_submit in a shell script, it will not find them,
> will it?
> 
> How must I tar all the output files (including the subdirectories) just
> after process completion, but just before moving the output files and
> removing the execute directory?
> 
> Thanks again,
> 
> Javier.
> 
> 
> 
> On Mon, 2008-02-04 at 15:46 +0000, Smith, Ian wrote:
> > I've used something similar under Condor-G which doesn't( didn't ?)
> > have
> >
> > this functionality. I create a time stamp file before the start of
> >
> > application then after running the app use
> >
> >
> >
> > $ find -newer time_stamp_file_name
> >
> >
> >
> > to work out which files have been created/modified and transfer them
> > back. If
> >
> > you use DAGman then a POST script could automatically unzip/untar the
> > output on the
> >
> > submit host to make things easier.
> >
> >
> >
> > -ian.
> >
> >
> >
> >
> >
> >
> >
> > From:condor-users-bounces@xxxxxxxxxxx
> > [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of carl langlois
> > Sent: 04 February 2008 14:57
> > To: Condor-Users Mail List
> > Subject: Re: [Condor-users] retrieving output subdirectories
> >
> >
> >
> >
> > Hi Javier,
> >
> >   I have the same problem as you, what i'm planning to do is wrap the
> > application in a .bat file or .sh if on unix and after the jobs is
> > complete juste zip everything and tell condor to retrived the zip
> > file.
> >
> >
> >
> > On Feb 4, 2008 5:37 AM, Javier Forment <jforment@xxxxxxxxxxxx> wrote:
> >
> > Hi, all... I'm trying to run an application in Condor which creates an
> > output subdirectory and write output files on it. I know that Condor
> > has
> > the feature of retrieving (and sending back to submit node) only the
> > output files created/modified in the plain execute directory at the
> > corresponding execute node.
> >
> > Is there any way to cope with this? I've browsed the archives at the
> > mailing list, but I only find messages telling that it is a well-known
> > issue, and the solution is to modify the application or writing a
> > custom
> > wrapper for condor to tar (and maybe gzip) the output
> > subdirectory/ies.
> >
> > Since I cannot modify the application, where can I find documentation
> > to
> > write such a custrom wrapper for condor?
> >
> > Thanks a lot,
> >
> > Javier.
> >
> >
> > _______________________________________________
> > Condor-users mailing list
> > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
> > with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/condor-users/
> >
> >
> >
> >
> >
> > _______________________________________________
> > Condor-users mailing list
> > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/condor-users/
> 
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/