Re: [HTCondor-users] APPEND_REQUIREMENTS vs JOB_TRANSFORM

[John M Knoeller <johnkn@xxxxxxxxxxx>]

You are correct,  APPEND_REQUIREMENTS is implemented client side in the code that turns a submit description into a job classad.

 

Most of the knobs that influence job submission are client side - which is why it is nearly impossible to create a fully correct job classad without using either condor_submit or the python Submit object.    The python Submit object and condor_submit both use the same c++ library code to generate job classads from a submit description, so they will generally work the same and respond to config in the same way.  This logic is complex and there is no hope of replicating it outside of HTCondor â it tends to change with every version of HTCondor as new capabilities are added.

 

JOB_TRANSFORM_* and SUBMIT_REQUIREMENT_* are implemented in the Schedd, everything else is in the client.

 

I would strongly urge you to throw out your code that creates job classads from scratch, and switch to using the python Submit object to create those classads.

 

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Frederic Boismenu via HTCondor-users
Sent: Monday, April 4, 2022 11:18 AM
To: htcondor-users@xxxxxxxxxxx
Cc: frederic.boismenu@xxxxxxxxxxx
Subject: [HTCondor-users] APPEND_REQUIREMENTS vs JOB_TRANSFORM

 

â
Hello HTCondor Users,

We are investigating different ways to "inject" a simple requirement clause into every users' job, something like "My.WorkerAttribute =?= Target.WorkerAttribute".

Our first attempt was to settle on APPEND_REQUIREMENTS which is attractive because it seems to be a less CPU intensive approach for our submit boxes.

It all worked well, until we tried to submit jobs directly as classad with Python binding. We are building the job definition as a classad.ClassAd(), 
which is then htcondor.Schedd().submit() 'ed.

To our surprise the APPEND_REQUIREMENTS does not seem to be honored in this case. But if we submit a job defined as htcondor.Submit() it is honored.

According to our tests, job transforms always works whether submitting from a classad or from a Submit object.

Even more surprisingly, while testing different configurations we noticed the APPEND_REQUIREMENTS behavior was retained by our client Python REPL even after restarting condor daemons,
which makes me suspect APPEND_REQUIREMENTS is actually implemented on client/library side at the job description parsing time, and that condor_schedd daemon is not even aware of it?

Could you please shade some light on the job submission actually works?  when and where macros are evaluated (such as $(Cluster)) ?
What's the relationship between the high level description level submit file format and actual classad sent to the schedd ? It seems there's a lot of magic actually happening in the client actually?

A diagram like the following which describing step 2. in details would be awesome :) 
http://research.cs.wisc.edu/htcondor/CondorWeek2009/condor_presentations/admin_tutorial/â


Many thanks in advance,
Fred