Re: [HTCondor-users] change to condor_submit - user feedback desired! (was Re: multiple condor_submit's - one cluster)

[Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>]

On 02/10/2015 03:12 PM, Brian Bockelman wrote:
> 
>> On Feb 10, 2015, at 2:59 PM, Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>
>> wrote:
>> 
>> On 02/10/2015 02:45 PM, Todd Tannenbaum wrote:
>> 
>>> input = in.$(process) queue 1000 The above requires the system to
>>> perform just one fork/exec, create just one network connection,
>>> perform authentication (schedd authenticate the condor_submit
>>> user) once, perform just one fsync to disk, etc.  On the other
>>> hand, invoking condor_submit 1000 times with "queue 1" results
>>> in 1000 fsyncs, 1000 authentications, etc.
>> 
>> So what does condor_submit_dag do for a dag w/ 1000 jobs (and no 
>> parent-child deps)?
> 
> It invokes condor_submit 1,000 times -
...

> DAGMan isn't particularly efficient here - on the other hand, it has
> an internal layer of consistency and queueing so it doesn't have to
> be!

So backport that "internal layer" to condor_submit and it won't have to
be efficient either. Or fix condor_submit_dag so it doesn't do 1K
fork/exec/fsync/ldap queries.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Attachment: signature.asc
Description: OpenPGP digital signature