[Condor-users] Running multiple batch files under one description file or running multiple description files under the same condor_submit command

["Alas, Alex [FEDI]" <aalas@xxxxxxxxxxxxx>]

Hello to all Condor users (Gurus, Experts, not so experts and the rest of us) again,

I have a very important question, I have a condor pool of 23 Windows nodes (CPU’s). The way we are running batching jobs today is creating a batch file that will encompass 500 process into one file. The way I understand condor when a submitting machine sends a job to the Central Manager Systems, it will allocate that batch file (submitting job) to only one CPU or the first CPU that matches its requirements, then the second, the third and so on until it matches the amount of time you stipulated to run the job in your queue. For a better understanding of my question I am adding a sample of the submitting job and a sample of the batch file we are running. As I said before some time we run some jobs with up to 500 process on one batch file. To clarify my question I will create two different scenarios, case 1 and 2;

Case 1: I need to know if there is a way I can run multiple batch files under the same description file for example adding the another testjob1.bat right after \\Sharename\Subdir1\Subdir2\testjob.bat; \\Sharename\Subdir1\Subdir2\testjob1.bat, in some cases up to 500 different batch files, and can those multiple and different files being executed on different CPU’s.

 

Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Case 1:

Description File sample:

UNIVERSE = VANILLA

REQUIREMENTS = (Arch == "INTEL" && OpSys == "WINNT52" || Arch == "INTEL" && OpSys == "WINNT51")

NOTIFY_USER = user1@xxxxxxxxxx,user2@xxxxxxxxxx

INITIALDIR = c:\condor\execute

SHOULD_TRANSFER_FILES = YES

WHEN_TO_TRANSFER_OUTPUT = ON_EXIT

TRANSFER_INPUT_FILES = \\Sharename\Subdir1\BatchProcessingPackage\Subdir3\Subdir4\lasEnvelop.exe

RUN_AS_OWNER = TRUE

EXECUTABLE =  \\Sharename\Subdir1\Subdir2\testjob.bat,  ||

  \\Sharename\Subdir1\Subdir2\testjob1.bat ||

  \\Sharename\Subdir1\Subdir2\testjob2.bat ||

  \\Sharename\Subdir1\Subdir2\testjob3.bat

OUTPUT = testjob.out.$(Process)

ERROR = testjob.err.$(Process)

LOG = testjob.log

QUEUE 1

Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Case 2:

If that is not possible is there a way to submit multiple jobs under one condor_submit command.

Condor_submit Test_job.txt, Test_job1.txt, Test_job2.txt… up to n jobs…

 

Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Samples:

Description File sample; Test_job.txt:

UNIVERSE = VANILLA

REQUIREMENTS = (Arch == "INTEL" && OpSys == "WINNT52" || Arch == "INTEL" && OpSys == "WINNT51")

NOTIFY_USER = user1@xxxxxxxxxx,user2@xxxxxxxxxx

INITIALDIR = c:\condor\execute

SHOULD_TRANSFER_FILES = YES

WHEN_TO_TRANSFER_OUTPUT = ON_EXIT

TRANSFER_INPUT_FILES = \\Sharename\Subdir1\BatchProcessingPackage\Subdir3\Subdir4\lasEnvelop.exe

RUN_AS_OWNER = TRUE

EXECUTABLE =  \\Sharename\Subdir1\Subdir2\testjob.bat

OUTPUT = testjob.out.$(Process)

ERROR = testjob.err.$(Process)

LOG = testjob.log

QUEUE 1

 

Testjob.bat:

echo off

\\Sharename\Subdir1\BatchProcessingPackage_beta\Subdir3\Subdir4\Envelop.exe 3 -1 2 \\Sharename\Subdir1\Subdir2\Envelop.txt \\Sharename\Subdir1\Subdir2 a.LAS (Process 1)

\\Sharename\Subdir1\BatchProcessingPackage_beta\Subdir3\Subdir4\Envelop.exe 3 -1 2  \\Sharename\Subdir1\Subdir2\Envelop.txt \\Sharename\Subdir1\Subdir2 b.LAS (Process 2)

 

 

Respectfully,

Alex Alas

Systems Administrator
Fugro EarthData Inc.

Tel. 301-948-8550 x219 Fax 301-963-2064 E-mail: aalas@xxxxxxxxxxxxx

7320 Executive Way, Frederick, MD  21704

Website: http://www.fugroearthdata.com