Re: [HTCondor-users] condor and AWS

["Kandes, Martin" <mkandes@xxxxxxxx>]

Hi Taha,

The latest version of condor_annex is being built into the HTCondor 8.7 development series by Todd Miller of the HTCondor team. You can view some of the latest updates he discussed this week at the Open Science Grid All-Hands Meeting here [1]. 

It is my understanding that for the most part this is still a manual process. i.e., if you are the administering the condor pool yourself, you can expand the pool to AWS on the fly as needed to meet user job demands. However, the use case here is easiest if you don't mind sharing the AWS resources across your users.

I've been evaluating the original condor_annex prototype and how the Open Science Grid may integrate its capabilities to support our users. You can see some overview of that evaluation here [2]. 

The largest challenge we face with the prototype (and probably still the latest version in HTCondor 8.7) is how to support multiple users with multiple AWS accounts all within the same pool. i.e., No one will want other people charging their credit card, obviously. 

I'm currently in the process of attempting to modify the original condor_annex prototype to use SSL certificates instead of the pool password for authentication. This will at least allow us to hand out and whitelist (or blacklist) user's individual AMIs that are trying to connect back to the pool to run their jobs. Users will then be required to drop in their Owner (username) into a startd attribute on worker nodes to only allow their jobs to run on their paid-for instances. 

If interested in experimenting with this modified version of the prototype, you can follow here [3] [4]. I expect to have most of these modifications sorted out by the end of the month or so with updated documentation. But again, it's all still going to be very much a prototype. 

Marty Kandes
UCSD

[1]

See link to pptx slide on the right-hand side. 

https://indico.fnal.gov/contributionDisplay.py?sessionId=16&contribId=16&confId=12973

[2]

See link to pdf at bottom. 

https://indico.cern.ch/event/531810/contributions/2311410/

[3]

http://www.t2.ucsd.edu/twiki2/bin/view/UCSDTier2/Condor_annex

[4]

https://github.com/mkandes/condor_annex

________________________________________
From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of Michael Pelletier [Michael.V.Pelletier@xxxxxxxxxxxx]
Sent: Friday, March 10, 2017 12:45 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] condor and AWS

TV,

Ben Cotton of Cycle Computing is a regular attendee at HTCondor Week, and I've been impressed with what I've seen of their product. As I understand it one of my company's divisions is using it for a certain project. I recommend you check the last couple of years of HTCondor Week presentations to find the "Better Solutions, Faster" presentations he's made.

They're doing quite interesting and exciting work, and they have additional capabilities that help optimize your use of cloud-based resources, such as a tool that can test your application against the various AWS instance types and availability zones to find the best bang for the buck, and what looks like a pretty good GUI.

        -Michael Pelletier.

> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On
> Behalf Of Taha Vasi
> Sent: Friday, March 10, 2017 3:37 PM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] condor and AWS
>
> Thx B... will certainly check out your talk that Steven send earlier and will
> check out condor_annex.
>
> I was looking at a product from Cycle computing, any opinion on using a
> commercial product versus  rolling your own ? or condor_annex ?
>
> Thanx
> -tv
>

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/