Re: [Condor-users] condor submit and dag man

[Matt Hope <Matt.Hope@xxxxxxxxxxxxxxx>]

I previously wrote throttling code into my submission 'daemon' to avoid too many idle jobs in the queue for performance reasons.
Since condor 7.2.x I found this is actually more harmful (due to increase complexity in the daemon) and I removed this restriction without any problems.

If you simply don't want to see them educate your users on the -r command line switch (it will be much more efficient this way too)

Matt
 

-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Mag Gam
Sent: 09 April 2010 19:39
To: Condor-Users Mail List
Subject: Re: [Condor-users] condor submit and dag man

Yes, you can make the assumption of condor_submit and schedd.

Basically, I am submitting over 20,000 jobs per day to our pool.
Instead of people seeing 20k idle jobs I would like to have something
native in condor_submit of schedd which will say, only have this many
idle. I have been using DAGMan to control this now and it works very
well, but it would be nice to have it intereted into
schedd/condor_submit.


On Fri, Apr 9, 2010 at 2:35 PM, Erik Paulson <epaulson@xxxxxxxxxxxx> wrote:
> I doubt it.
>
> First off, I assume you mean the schedd when you say condor_submit.
>
> - Job dependencies are DAGMan's bread and butter, so for as long as
> there's a DAGMan it seems like there's no reason to duplicate that
> code in the schedd.
>
> - MaxRunning might make sense to have on a per-user breakdown in the
> schedd, or at least have a more fine-grained control that the
> schedd-wide MAX_JOBS_RUNNING config setting. If the job ad could refer
> to the queue then you could do this today, ie something like:
>   requirements = TARGET.OpSys == "LINUX" &&
> (count(localqueue.runningjobs) < 50)
> (to be clear, I just made that syntax up, so don't try it and expect it to work)
>
> -MaxIdle doesn't seem very useful for the schedd
>
> As far as emulating them with other tools, that's what DAGMan is for.
> What are you trying to do that doesn't work for you?
>
> -Erik
>
> On Fri, Apr 9, 2010 at 2:20 PM, Mag Gam <magawake@xxxxxxxxx> wrote:
>> Will Condor's DAGMan features ever be migrated to condor_submit? For
>> example, having job dependencies and MaxRun and MaxIdle options...
>>
>> If not, is it possible to emulate these features with other tools?
>>
>> TIA
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/condor-users/
>>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/

----
Gloucester Research Limited believes the information provided herein is reliable. While every care has been taken to ensure accuracy, the information is furnished to the recipients with no warranty as to the completeness and accuracy of its contents and on condition that any errors or omissions shall not be made the basis for any claim, demand or cause for action.
The information in this email is intended only for the named recipient.  If you are not the intended recipient please notify us immediately and do not copy, distribute or take action based on this e-mail.
All messages sent to and from this email address will be logged by Gloucester Research Ltd and are subject to archival storage, monitoring, review and disclosure.
Gloucester Research Limited, 5th Floor, Whittington House, 19-30 Alfred Place, London WC1E 7EA.
Gloucester Research Limited is a company registered in England and Wales with company number 04267560.
----