[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Queuing N items from file



On 8/27/2020 11:19 AM, Eremey Valetov wrote:
Hello All,

I am using a computing cluster where I am allowed to have a limited number of jobs in the queue.

I would like to queue $(njobs) at a time, each with parameters from one line of a file:

queue par1,par2,par3,par4,par5 from [0:$(njobs)] inputs.txt

Instead of doing a slice to ensure only njobs are simultaneously in the queue, I suggest just using MAX_MATERIALIZE in your submit file.
For example, say inputs.txt has 1,000,000 lines, but you are only permitted to have 500 jobs in the queue at once.  You could do:

   max_materialize = 500
   queue x,y,z from inputs.txt

The above ensure the number of jobs in the queue for this cluster never exceeds 500... as soon as a job leaves the queue (because it completed or was removed), the next job in the cluster will be "materialized" and show up as idle.    E.g. upon submission, you may get jobs 50.0 thru 50.499... as soon as job 50.0 completes, job 50.500 will get submitted.  To remove the entire cluster of jobs (i.e. remove all jobs and do not materialize any more), do "condor_rm <cluster_id>", e.g. do "condor_rm 50" instead of "condor_rm 50.5". 

But re your specific question/issue below, it would help to see how you are setting $(njobs) via the command-line.  In other words, what exactly are you typing on the command line?  It may also be useful to see the output from "condor_version".

Thanks
Todd



However, this is treated as if the slice were [0:], even though njobs is set using a command-line argument to condor_submit. On the other hand, the following works as expected:

queue par1,par2,par3,par4,par5 from [0:1000] inputs.txt

Is there perhaps a way to have a variable in the slice, as in [0:$(njobs)], so that it works correctly?

Thanks,
Eremey Valetov




_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/


-- 
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685