Re: [HTCondor-users] Managing dynamic services

[Greg Thain <gthain@xxxxxxxxxxx>]

On 11/16/18 6:57 PM, Stuart Anderson wrote:
> Is there a standard way for Condor to manage local privileged dynamic services on an execute machine depending on whether any currently matched jobs have requested it?
>
> Put another way, instead of matching a job to a machine that already has a service running, what is the best way to handle individual jobs optionally requesting a service that is started by condor when a match occurs and before the condor job starts, along with Condor stopping the service after the last job requesting it has finished running?
>
> The context of this question is Linux systems running systemd if there is a natural hook there.


Stuart:


This is an interesting question, and I think there's at least one way to 
do this with HTCondor, but I have a couple of questions to put a finer 
point on potential solutions

Usually, pools would be configured so this service would be started at 
boot time on all machine, but HTCondor would limit the number of running 
jobs that use the service using some mechanism. I assume that you don't 
want to have the service always running because it uses a non-trivial 
amount of resources, even when idle. If so, we may want to reflect this 
additional machine load in the startd. Also, it seems that many jobs on 
the same machine from different users can use this service without too 
much additional load on the service. Do you want the service to get 
killed when the last job finishes, or is it ok to have some "grace 
time", when no jobs are running, in order to encourage a subsequent job 
to start?

Also, is it sufficient for Condor to start the service before starting 
the first job, or do we need to block the job from starting until the 
service is "ready"?

Thanks,


-greg