Re: [HTCondor-users] job_transform and preserving requirements

[John M Knoeller <johnkn@xxxxxxxxxxx>]

Hi. Stefano.  You are on the right track.  Schedd JOB_TRANSFORMs is perfect for this.

I see that Jason has already answered this.  His answer is good (see the update)
I just want to chime in with a bit of explanation. 

Because you want to *modify* an expression in a job,  it's better to use the native syntax
rather than the job router syntax.  The job router syntax puts the transform into a classad
which is why it is enclosed with [], and has ; between statements.  The job router syntax
can set job attributes, but it can't really modify them.

The native syntax is more like the config or submit language, it can use $() expansions.  
The native syntax works in 8.8 Schedd, but it was not documented until 8.9.  Syntax is
documented here https://htcondor.readthedocs.io/en/latest/misc-concepts/transforms.html

The native syntax can use also use config if statements, which can be helpful when you want to modify expressions cleanly

For your use case, your transform would look something like this

JOB_TRANSFORM_NAMES = $(JOB_TRANSFORM_NAMES) Stor_ams
 
JOB_TRANSFORM_Stor_ams @=end 
    REQUIREMENTS  AcctGroup == "ams"
    if defined My.Requirements
        SET Requirements TARGET.GPFS_AMS && ( $(My.Requirements) ) 
    else
        SET Requirements TARGET.GPFS_AMS 
    endif
@end

A couple of points here.  

I use == rather than =?= for the AcctGroup comparison, since if AcctGroup is undefined, the transform should not match.

$(My.Requirements) will expand to the job's current requirements expression,  and it can also be used in an if
statement so that if the requirements statement is empty, we don't end up creating an invalid expression.   This is nearly impossible for a job, so I think you can safely use the example that Jason sent you.  I just want to show how it would work.

I Also I don't compare TARGET.GPFS_AMS to false, since once again, if it is undefined, we don't want to match.
This is a bit safer because  if TARGET.GPFS_AMS  is 0,  That would *evaluate* to false,  but it would not =!= false. 

-tj

-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Stefano Dal Pra
Sent: Tuesday, July 7, 2020 1:23 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] job_transform and preserving requirements

Hello,
htcondor 8.8.9 speaking.

We have several local user groups, each one using a specific filesystem 
which is expected to be mounted and available on the execute node.
Example:
-  jobs submitted by users in the "ams" unix group would work with 
/storage/gpfs_ams ;
-  jobs submitted by users in the "borexino", "pamela", "rdfa",..., 
groups would work with /storage/gpfs_data
-  jobs submitted by users in the  ..., ..., ..., groups would work with 
/storage/gpfs_<xyz>

I want to add requirements to these jobs submitted to one schedd, so 
that they require a machine where the needed

filesystem is available, which is advertised by these custom classads:
GPFS_AMS  --> true / false / undefined
GPFS_DATA --> true / false / undefined
...
GPFS_<xyz> --> true / false / undefined

My idea would be to use JOB_TRANSFORM somewhat like this:

#this should set AcctGroup
use FEATURE:AssignAccountingGroup($(T1_SHARED_SCRIPT_DIR)/Hgroups.txt)

JOB_TRANSFORM_NAMES = $(JOB_TRANSFORM_NAMES) Stor_ams Stor_data 
Stor_name1 Stor_name2
JOB_TRANSFORM_Stor_ams = [ Requirements = AcctGroup =?= "ams";\
set_requirements = (TARGET.GPFS_AMS =!= False) ]
JOB_TRANSFORM_Stor_data = [ Requirements = 
StringListMember(AcctGroup,"borexino:pamela:rdfa:<SNIP>",":");\
set_requirements = (TARGET.GPFS_DATA =!= False)]

JOB_TRANSFORM_name1 = [ ....]
JOB_TRANSFORM_name2 = [ ....]

with the problem however that doing so i would override the original 
requirements specified by the users.
Is there a way to preserve them?

I see that there is the knob: APPEND_REQUIREMENTS
and i probably could set that using a chain of nested IfThenElse( ..., 
..., ...). However i would prefere JOB_TRANSFORM because it seems more 
versatile, as i could later add
more constraints if needed (i.e. a runtime limit or memory limit per 
group, or so)

Thanks for any suggestion

Stefano
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/