Re: [HTCondor-users] Preventing users from submitting too many jobs

["Krieger, Donald N." <kriegerd@xxxxxxxx>]

Hi Vlad,

The use of condor_q in this way is problematic since it's a relatively heavy-weight call whose use should be minimized.
This problem is exacerbated by the fact that the conditions you want to catch are ones in which there are large numbers of jobs in the queue.
So the condition under which the your check is important is one in which the call will be more expensive.

We handle the load of calls to condor_q by including a call to it in a single process which runs the call every 3 minutes.
The output is sent to /dev/shm/condor_q_cachefile .
That file can be accessed, searched, etc. indiscriminately without loading the system.

Regards,
 
Don
 

Don Krieger, Ph.D.
Department of Neurological Surgery
University of Pittsburgh

> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf
> Of Vladimir Brik
> Sent: Wednesday, May 27, 2015 9:30 AM
> To: HTCondor-Users Mail List
> Subject: [HTCondor-users] Preventing users from submitting too many jobs
> 
> Hello,
> 
> I would like to have a mechanism to prevent a user from submitting new jobs if
> he or she already has a large number of jobs in the queue (e.g.
>  >20,000).
> 
> SUBMIT_MAX_PROCS_IN_CLUSTER and MAX_JOBS_SUBMITTED help, but don't
> solve the problem completely because they are global limits, and I'd like to
> have per-user limits.
> 
> What I am thinking of doing is replacing condor_submit with a wrapper that
> runs condor_q to check if the user submitted too many jobs, and if so, drops a
> config file with appropriate SUBMIT_REQUIREMENT_* to block the user from
> submitting more. The wrapper would be clever enough to skip condor_q if its
> called too frequently.
> 
> Does anybody know of a simpler way to have per-user limits on the number of
> jobs in queue?
> 
> Is there a way to do something similar for jobs submitted using python bindings
> and/or remote submissions?
> 
> 
> Thanks,
> 
> Vlad
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/