Re: [HTCondor-users] HTCondor Cluster within Slurm Job

[Michael Pelletier <michael.v.pelletier@xxxxxxxxxxxx>]

Iâm interested in this idea as well - the bulk of the HPC in the combined companies is now Slurm, and thatâs intended to be the supported standard (alas), so having a bridge capability would be useful.

 

It seems like itâd be some sort of variant of HTCondor-CE with âpilotâ jobs.

 

Michael V Pelletier

Principal Engineer

Raytheon Technologies

Digital Technology

HPC Support Team

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Leslie Hart - NOAA Federal via HTCondor-users
Sent: Wednesday, September 29, 2021 3:49 PM
To: htcondor-users@xxxxxxxxxxx
Cc: Leslie Hart - NOAA Federal <leslie.b.hart@xxxxxxxx>
Subject: [External] [HTCondor-users] HTCondor Cluster within Slurm Job

 

Hi,

 

Is it possible (and is there an existing recipe) to start up a "private" HTCondor Cluster within a Slurm job. We have users who would like to allocate a number of nodes and then use those nodes as an HT cluster for the duration of the job. Ideally, we could supply a few commands that they would use at the beginning and end of their Slurm batch job to start and shutdown the cluster (the middle would be comprised of a series of HTCondor jobs, of course. e.g. HTCondorStart (would figure out the nodes that Slurm has allocated and create the cluster). HTCondorWait (would wait until all HTCondor jobs completer) and HTCondorFinish (would gracefully shut down HTCondor).

Thanks,

Leslie Hart