Re: [Condor-users] Can a job send a trigger to let other jobs start?

[Carsten Aulbert <carsten.aulbert@xxxxxxxxxx>]

Hi Matt,

On Tuesday 15 December 2009 11:53:04 Matt Hope wrote:
> The problem with trying to do it this way is that it does not play well
>  with pre-empted jobs restarting
> 
So far, we did not need pre-emption usually we have enough slots available.

> I personally would break this down into two independent jobs, one of which
>  stages the data (to a network share) the other of which does the
>  processing.
> 

The problem is the data is already close, i.e. just 2-3 switch hops away on a  
network share which can serve 600 MByte/s continuously unless you ask it to 
perform too many disk seeks.

> 
> If you get fancy you can likely try to stage the data to the local disk and
>  use requirements rewriting to force the follow up jobs to go to the same
>  machine but having it on the network is much simpler if your network can
>  handle a few gigs transfer (in a compressed, or at least tarballed form
>  will significantly reduce the transfer overhead).
> 
These are already nice 120 MByte files internally compressed IIRC, so staging 
them to the local disk might help a bit, but possibly not much.

Thanks for the hints

Carsten