Re: [HTCondor-users] change to condor_submit - user feedback desired! (was Re: multiple condor_submit's - one cluster)

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 2/10/2015 2:07 PM, Dimitri Maziuk wrote:
> On 02/10/2015 01:43 PM, Todd Tannenbaum wrote:
>
> > ...  I
> > think I will focus first on the details for the ability of condor_submit
> > to scan the filesystem and do a submit for each file
>
> I still don't get why not just use VARS from condor_submit_dag: it seems
> to me that VARS substitution, other than $(cluster) and $(process) is
> what's really missing here -- and even then there's $ENV(). Once there's
> VARS, a condor_submit_many wrapper that scans the filesystem should take
> about 5 minutes to write. Or I am missing the point entirely, I do that
> often enough too.

Conceptually you are correct.  However, consider when you do something like
    input = in.$(process)
    queue 1000
The above requires the system to perform just one fork/exec, create just 
one network connection, perform authentication (schedd authenticate the 
condor_submit user) once, perform just one fsync to disk, etc.  On the 
other hand, invoking condor_submit 1000 times with "queue 1" results in 
1000 fsyncs, 1000 authentications, etc.
Doing any kind of wrapper that invokes condor_submit (in its current 
form) for each single job would be unfortunate from a scalability 
perspective...

thanks
Todd

> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/


--
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685