[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Condor with Ec2



That's a good overview. We use the condor_job_router, which now provides some of its own hook call-outs, and its routing rules. The hooks use some of Amazon's Web Services (e.g. SQS & S3) to pass the job information to a newly created EC2 job, which is the result of the routing.

Best,


matt

William Henry wrote:
Hi Jon,

This is what we've been doing here at Red Hat.  We have an external agent called the jobrouter and the magic is in its routing capability. We implemented new hooks (i.e. job hooks) to assist in doing this. These new hooks in the job router are similar to the hooks available in the startd and starter.  It also takes advantage of AWS Web services to transfer data in and out.

I've not been the one working on this directly. I've just been doing some testing and playing with it. So I expect someone like Matt will chime in on the topic.

William Henry
Red Hat, MRG (http://www.redhat.com/mrg)
Colorado, USA.  +1 719  302 2302

----- "Jonathan D. Proulx" <jon@xxxxxxxxxxxxx> wrote:

Hi William,

I for one am interested in more details of your aproach.

Getting our Condor cluster to dynamicly expand into EC2 space has
been
in my queue for a while, and may finally be getting back to the top.

My thoughts such as they've been is to have an external agent that
watches the queue and determines (through magic?) when new AMIs
should
be spawned ( i'm thinking shutdown logic should probably live on the
AMI so it's tied to the billing cycle ), with the AMI mostly a
generic
resource like our other cluster nodes (using GCB for communication).

-Jon


On Wed, Dec 03, 2008 at 07:41:47PM -0500, William Henry wrote:
:
:FYI,
:
:I posted a couple of articles about using Condor with EC2:
:
:First just using Condor to start an AMI:
:http://ipbabble.com/2008/11/mrg-grid-and-ec2.html
:
:Second using Condor and some new hooks to schedule jobs in an AMI in
EC2:
:http://ipbabble.com/2008/12/mrg-and-ec2-enhanced.html
:
:Not all the details are here. There are some settings needed for the
Enhanced (2nd option).
:
:Let me know if any of you would like those or if you think I should
post a more exhaustive example on the blog.
:
:
:Best,
:William Henry
:Red Hat, MRG (http://www.redhat.com/mrg)
:_______________________________________________
:Condor-users mailing list
:To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
with a
:subject: Unsubscribe
:You can also unsubscribe by visiting
:https://lists.cs.wisc.edu/mailman/listinfo/condor-users
:
:The archives can be found at: :https://lists.cs.wisc.edu/archive/condor-users/
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at: https://lists.cs.wisc.edu/archive/condor-users/
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at: https://lists.cs.wisc.edu/archive/condor-users/