Re: [HTCondor-users] Submission to remote slurm cluster is Failing consistently

Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab

The SLURM partition can be specified in the htcondor job routing via the set_default_queue parameter (see below):

JOB_ROUTER_ENTRIES @=jre

[

GridResource = "batch slurm";

TargetUniverse = 9;

name = "GRID jobs";

set_default_queue = "grid";

]

@jre

AFAIK there is no way to specify the âqos in the job routing. One way would be to patch the /usr/libexec/condor/glite/bin/slurm_submit.sh blahp script and add it there.

However you can define a DefaultQOS on account/association level using sacctmgr which will be used when you donât define a âqos during submission:

sacctmgr modify account myAccount set DefaultQOS='grid

The above statement only works if the account has access to the QOS. You can set this with: sacctmgr modify account myaccount set QOS='debug,gridâ

You must also make sure that the partition allows the QOS.

This is specified in /etc/slurm/slurm.conf on the controller.

You can check this with: scontrol show part

Check the âAllowQosâ parameter in the output if the QOS is included or the value is set to âALLâ

Hope this helps

Best

Ãmit Seren Msc

HPC Engineer

+4369910269552

Vienna BioCenter (GMI, IMP, IMBA)

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of "hasanbaigg@xxxxxxxxx" <hasanbaigg@xxxxxxxxx>
Reply-To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Date: Tuesday, 2. February 2021 at 20:26
To: Jaime Frey <jfrey@xxxxxxxxxxx>
Cc: bosco-discuss <bosco-discuss@xxxxxxxxxxxxxxxxxxx>, HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Submission to remote slurm cluster is Failing consistently

I am not really sure how to do that. I will explore how to run a job directly to slurm and will get back to you with the output.

Meanwhile, if you know any relevant documentation, kindly share.

thanks

regards

Hasan

On Feb 2, 2021, at 1:53 PM, Jaime Frey <jfrey@xxxxxxxxxxx> wrote:

BOSCO and HTCondor donât use the -q/--qos command-line option when submitting a job to Slurm. There is a way to have them set the -p/--partition option, but the bosco_cluster tool doesnât use it when testing a new setup.

Can you try submitting a job directly to Slurm on the cluster login node without using the -p or -q options of sbatch?

- Jaime

On Feb 2, 2021, at 12:43 PM, hasanbaigg@xxxxxxxxx wrote:

Thanks for your concerns,

I will look into the qos documentation. Meanwhile, would expect any SLURM expert to chip in.

regards

Hasan

On Feb 2, 2021, at 1:32 PM, christoph.beyer@xxxxxxx wrote:

Uh,

hopefully someone with slurm experience can jump in here (I better stick to my alternative HTC facts) :D :D :D

As far as I know the partition you use is defined with a list of possible qos (default,debug) for ex. in the submit you can request the qos with the '-q' option.

scontrol show part <partname>

It looks like your job coming from the HTC side of things is requesting a qos that is not defined for the partition you want to use.

Unfortunately that is even thinner ice for me than usualy on this list :(

Also I did not fully get the concept of slurm at the time and started using condor instead ;)

But you seem to be on the right track, and should check the documentation about qos and how to define them for partiions and users alike:

https://slurm.schedmd.com/qos.html#partition

Best

christoph

--
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx

Von: hasanbaigg@xxxxxxxxx
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
CC: "bosco-discuss" <bosco-discuss@xxxxxxxxxxxxxxxxxxx>
Gesendet: Dienstag, 2. Februar 2021 18:53:07
Betreff: Re: [HTCondor-users] Submission to remote slurm cluster is Failing consistently

Hi,

Now I ran it on the server (where bosco node is hosted) and it shows me the following output

User Def Acct Admin Cluster Account Partition Share MaxJobs MaxNodes MaxCPUs MaxSubmit MaxWall MaxCPUMins QOS Def QOS

---------- ---------- --------- ---------- ---------- ---------- --------- ------- -------- -------- --------- ----------- ----------- -------------------- ---------

hbaig pi-mendes None xanadu pi-mendes 1 general,himem,speci+

-bash-4.2$

Could you please tell me if it tells something meaningful which needs to be fixed.

regards

Hasan

On Feb 2, 2021, at 12:37 PM, christoph.beyer@xxxxxxx wrote:

acctmgr show user <user> withassoc

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

Mailing List Archives

Public Access

Re: [HTCondor-users] Submission to remote slurm cluster is Failing consistently