[HTCondor-users] Application specific scheduler

[Nick Cooper <nick.cooper@xxxxxxxxxxxxxxxxxxxx>]

Hi All,

I am currently looking at migrating from our home grown distributed computing software toÂHTCondor. Over the years, user have created complex "job managers" written in C++ which are equivalent to application specificÂDAGManÂscripts. To reduce the burden on users migrating toÂHTCondorÂwe would like to provide an adaptor between a "job manager" andÂHTCondor.

An example of a simple "Job Manager" is one which (all within the same cluster):

1. Requests 1000 simulation jobs to be executed

2. When all 1000 simulation jobs are completed, creates a database and loads the results into it

3. Does analysis on the results in the database and based on the analysis requests further simulationÂjobs to be executed. All without any user involvement.

From what I have read our options are:

1.ÂWeb Service: Write an adapter using the SOAP interface. I suspect there is not enough feedback regarding when a job completes / fails.

2.ÂDAGMan: Write an adapter that generatesÂDAGManÂscripts.

3.ÂDRMAA: Write an adapter that submits and monitors jobs via theÂDRMAAÂAPI.

Can someone confirm if I am one the correct track?Â

Does anyone have any suggestions / words of wisdom for this kind of requirement?

Further info:

- Windows based pool

- Job manager is a C++ÂDLL

- Looking at using the current stable release ofÂHTCondor

- Jobs will run in the Vanilla Universe

- Jobs will need to be run under the submitters Active Directory credentials

Thanks Nick