Re: [Condor-users] Condor with AMQP

["Berg, Allen" <aberg@xxxxxxxx>]

Matt

I am defiantly interested where can I find out more information on Rob Rati's implementation?

We have a client that submits jobs from windows to the condor cluster.  The cluster is a all Linux machines RHEL 5.4.
What I learned this morning is that Ganglia was actually the root cause of the problem.  I moved the master to a machine that did not have Ganglia and the RRDtool installed, and everything seems to work fine. 

We are currently using a single generic queue for all the machines without any classads and using job hooks.

What I still find interesting is if I submit say 150 sleep jobs and I have 56 nodes available it seems that say 46 nodes will take off and run a batch then the number or running nodes drops to about half the started nodes then it will drop again until all the jobs complete the number of nodes continually declines in usage.

It just seems to be odd behavior I would expect all nodes to pick up and start working until all jobs were completed.

Thanks
Allen

-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Matthew Farrellee
Sent: Tuesday, September 21, 2010 10:42 AM
To: Condor-Users Mail List
Subject: Re: [Condor-users] Condor with AMQP

Allen,

Are you using the Startd's work fetch (/job hooks) functionality to pull 
work from the messaging queues?

If so, you should be looking at the execute nodes to see what the 
problem may be. It might be helpful to describe the mechanism you're 
using to deliver the work to Startds some more. Rob Rati implemented 
just such a system on top of Qpid, which you might be interested in.

Best,


matt

On 09/20/2010 04:17 PM, Shahaan Ayyub wrote:
>
> Allen,
> I have never worked with Qpid but it seems from having a quick look at
> the documentation that it simply provides a high level interface, in
> your case, to condor. I am amazed as to why native condor commands are
> not working? Otherwise you might have to look for a wrapper around
> native condor commands.
> Sorry couldn't be of much help to you.
> Regards,
> Shahaan
> On 21/09/2010, at 5:03 AM, Shahaan Ayyub <shahaan@xxxxxxxxx
> <mailto:shahaan@xxxxxxxxx>> wrote:
>
>> Hi Allen,
>> What does condor_q -better-analyze say for different timestamps, i.e.
>> when some of the jobs are held whilst some of them are still
>> running/completed.
>>
>> Regards,
>> Shahaan
>>
>> On 21/09/2010, at 3:07 AM, "Berg, Allen" <
>> <mailto:aberg@xxxxxxxx>aberg@xxxxxxxx <mailto:aberg@xxxxxxxx>> wrote:
>>
>>> We have a relatively small condor cluster its fifteen machines with a
>>> total of 140 cpus.
>>>
>>> We have implemented it using Apache Qpid Daemon is installed on the
>>> master node. This package provides the queue “server”. It is the
>>> facility that provides message queuing to the cluster. The Apache
>>> Qpid API for C++ is installed on each cluster node.
>>>
>>> What I am seeing that I have questions about is that when I submit
>>> say two jobs very simple just a sleep command for two of the nodes.
>>> The first job will take off and run, the second job will sit there
>>> for possibly 20 minutes before it times out. Within any of the condor
>>> logs I am not seeing any errors or any indications of weirdness. Then
>>> if I run a larger test of say 40 jobs to sleep for 5 seconds, I would
>>> expect that when I send the 40 jobs in they would all be picked up
>>> and run completing in a reasonable amount of time. What I really see
>>> is maybe 20 jobs take off, then 12 will start then maybe 8 and the
>>> last few will complete. How can I find/learn out how the queue
>>> actually performing and what can I do to better tune the queue.
>>>
>>> Thanks
>>>
>>> Allen
>>>
>>> This message and any enclosures are intended only for the addressee.  Please
>>> notify the sender by email if you are not the intended recipient.  If you are
>>> not the intended recipient, you may not use, copy, disclose, or distribute this
>>> message or its contents or enclosures to any other person and any such actions
>>> may be unlawful.  Ball reserves the right to monitor and review all messages
>>> and enclosures sent to or from this email address.
>>> _______________________________________________
>>> Condor-users mailing list
>>> To unsubscribe, send a message to
>>> <mailto:condor-users-request@xxxxxxxxxxx>condor-users-request@xxxxxxxxxxx
>>> <mailto:condor-users-request@xxxxxxxxxxx> with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> <https://lists.cs.wisc.edu/mailman/listinfo/condor-users>https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>> The archives can be found at:
>>> <https://lists.cs.wisc.edu/archive/condor-users/>https://lists.cs.wisc.edu/archive/condor-users/
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/

_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/



This message and any enclosures are intended only for the addressee.  Please  
notify the sender by email if you are not the intended recipient.  If you are  
not the intended recipient, you may not use, copy, disclose, or distribute this  
message or its contents or enclosures to any other person and any such actions  
may be unlawful.  Ball reserves the right to monitor and review all messages  
and enclosures sent to or from this email address.

Attachment: master.local
Description: master.local

Attachment: carod.config
Description: carod.config

Attachment: carod_hook.py
Description: carod_hook.py