Re: [HTCondor-users] condor and AWS

[Burt Holzman <burt@xxxxxxxx>]

Hi Taha,

We have been doing this with glideinWMS on top of HTCondor (see http://hepcloud.fnal.gov for details of our pilot project). Based on certain job characteristics (including idle jobs in the queue), we decide when to provision AWS resources - and join them into an existing on-premises pool. Weâve done demos where we also did this as a separate pool at AWS and GCE at large scale, provisioning the local resources via our grid
gateways.

You can also do this with condor_annex and your own demand-based algorithm. I know Brij Kashal in India has done something similar on MS Azure.

- B

> On Mar 10, 2017, at 8:16 AM, Taha Vasi <tvasi@xxxxxxxxxxxxx> wrote:
> 
>  
> Hi,
>  
> Has anyone implemented a condor pool, with bursting into AWS to eliminate peaks of jobs?
>  
> So the idea is to spin up machines in AWS when the job Q internally gets above a certain amount.. till the Q drops below some threshold.
>  
> Would it make sense to setup a separate condor pool in the cloud and then flock to it? Or just spin up AWS instances and have them join the existing pool whenever  you determine?
>  
> Thanx
> -tv
>  
> 
> Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorized and regulated by the Financial Conduct Authority.
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/