[HTCondor-users] Dynamic EC2 provisioning

[Tiago Macarios <tiagomacarios@xxxxxxxxx>]

Hi,

We have a HTCondor cluster which normally is enough for our needs, but from time to time the queue time gets too long. Since this is not a constant problem we were wondering if there is a way to have condor to dynamically start EC2 instances to increase the pool size.

I guess algorithm would be something like:

If queue size did not decrease in the last 5 hours:

    start an amazon instance

else

    if any instance is running:

        terminate one of them

From what I gather from the documentation this should be doable? I was wondering is someone already tried to do it and could share scripts or point me to the right direction.

It seems like the right way to do it would be to have condor to automatically submit an ec2 job if the conditions above were met and terminate/delete the job afterwards.

Cheers,

Tiago