Re: [HTCondor-users] JOB TRANSFORM Questions

[Chris Brew - UKRI STFC <chris.brew@xxxxxxxxxx>]

Thank you TJ, both methods work like a charm, now I just need to decide which one to use.

Yours,
Chris.

ïOn 16/10/2019, 22:05, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:

    Actually, now that I look at this more closely,  the other way to fix this is to use RequstedAcctGroup
    as the 3rd argument to the usermap function, and to duplicate the usermap function as the second argument
    in your strcat.  that would get rid of the ordering dependency. 
    
    JOB_TRANSFORM_AssignGroup @=end
    [
    copy_Owner="TestAcctGroupUser";
    copy_TestAcctGroup="RequestedAcctGroup";
    set_TestPartition = "group_LOCAL";
    eval_set_TestAcctGroup = usermap("Groups",TestAcctGroupUser, RequestedAcctGroup); 
    eval_set_TestAccountingGroup = strcat(TestPartition, ".", usermap("Groups",TestAcctGroupUser, RequestedAcctGroup), ".", TestAcctGroupUser);
    ]
    @end
    
    copy operations happen before set operations which happen before eval_set operations. 
    
    And since, you can use $() substitutions in your config files, you can do this make things a bit more readable.
    
    TEST_USERMAP_EXPR = usermap("Groups",TestAcctGroupUser, RequestedAcctGroup)
    
    JOB_TRANSFORM_AssignGroup @=end
    [
    copy_Owner="TestAcctGroupUser";
    copy_TestAcctGroup="RequestedAcctGroup";
    set_TestPartition = "group_LOCAL";
    eval_set_TestAcctGroup = $(TEST_USERMAP_EXPR); 
    eval_set_TestAccountingGroup = strcat(TestPartition, ".", $(TEST_USERMAP_EXPR), ".", TestAcctGroupUser);
    ]
    @end
    
    -tj
    
    -----Original Message-----
    From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of John M Knoeller
    Sent: Wednesday, October 16, 2019 3:12 PM
    To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
    Subject: Re: [HTCondor-users] JOB TRANSFORM Questions
    
    There isn't any need to use evalset for TestAcctGroup.  If you set it as an expression, it will still get set before
    the evalset of TestAccountingGroup, and will thus be defined. 
    
    That being said,  you may not want to shove an expression into the job.  In that case you need to control the order of
    the evalset statements.  You can control the order of evaluation by using the native transform syntax directly. 
    
    Have a look at the Schedd log on startup, and you will see that you transform is being converted
    into native syntax.    That syntax is evaluated top down, unlike the job-router style syntax that you
    are currently using. 
    
    If your transform does not start with [  then it will be parsed as the native syntax. So your config could look like this
    
    JOB_TRANSFORM_AssignGroup @=end
            NAME AssignGroup
            COPY Owner TestAcctGroupUser
            COPY TestAcctGroup RequestedAcctGroup
            SET TestPartition "group_LOCAL"
            EVALSET TestAcctGroup usermap("Groups",TestAcctGroupUser,TestAcctGroup)
            EVALSET TestAccountingGroup strcat(TestPartition, ".", TestAcctGroup, ".",  TestAcctGroupUser)
    @end
    
    The native syntax is described in a bit more detail by the condor_transform_ads tool.  run
    
    condor_transform_ads -help rules
    
    to see the rules. 
    
    -tj
    
    -----Original Message-----
    From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Chris Brew - UKRI STFC
    Sent: Wednesday, October 16, 2019 8:47 AM
    To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
    Subject: [HTCondor-users] JOB TRANSFORM Questions
    
    Hi,
    
    Iâm trying to our horrible nested IfThenElse statements that set up our hierarchical accounting groups with a nice clean JOB_TRANSFORM.
    
    Naively I started with:
    
     JOB_TRANSFORM_AssignGroup @=end
    [
    copy_Owner="TestAcctGroupUser";
    copy_TestAcctGroup="RequestedAcctGroup";
    set_TestPartition = "group_LOCAL";
    eval_set_TestAcctGroup = usermap("Groups",TestAcctGroupUser,TestAcctGroup); 
    eval_set_TestAccountingGroup = strcat(TestPartition, ".", TestAcctGroup, ".", TestAcctGroupUser);
    ]
    @end
    
    but TestAccountingGroup always got set to âundefinedâ because TestAcctGroup was âundefinedâ when it was evaluated.
    
    Iâve âsolvedâ it by setting TestAccountingGroup in a second transform, but is there a cleaner way of controlling which order the eval_sets get evaluated?
    
    That works to set TestAccountingGroup to the default for that user but if I try to select a non-default group I still get the default:
    
    The config is:
    
    SCHEDD_CLASSAD_USER_MAP_NAMES = $(SCHEDD_CLASSAD_USER_MAP_NAMES) Groups
    CLASSAD_USER_MAPFILE_Groups = /etc/condor/maps/local_user_map
    # Assign groups automatically
    JOB_TRANSFORM_NAMES = $(JOB_TRANSFORM_NAMES) AssignGroup AccountingGroup
    JOB_TRANSFORM_AssignGroup @=end
    [
    copy_Owner="TestAcctGroupUser";
    copy_TestAcctGroup="RequestedAcctGroup";
    set_TestPartition = "group_LOCAL";
    eval_set_TestAcctGroup = usermap("Groups",TestAcctGroupUser,TestAcctGroup);
    ]
    @end
    
    JOB_TRANSFORM_AccountingGroup @=end
    [
    eval_set_TestAccountingGroup = strcat(TestPartition, ".", TestAcctGroup, ".", TestAcctGroupUser);
    ]
    @end
    
    The map file has:
    
    $ grep brew /etc/condor/maps/local_user_map
     *  brew CMS,ATLAS
    
    The Job definition has:
    
    $ grep TestAcctGroup testjob
    +TestAcctGroup         = "ATLAS"
    
    But the eventual job classad has:
    
    RequestedAcctGroup = "ATLAS"
    TestAccountingGroup = "group_LOCAL.CMS.brew"
    TestAcctGroup = "CMS"
    TestAcctGroupUser = "brew"
    TestPartition = "group_LOCAL"
    
    Can anyone see what Iâve done wrong?
    
    Many Thanks,
    Chris.
    
    --
    Dr Chris Brew
    Scientific Computing Manager
    Particle Physics Department
    UKRI - STFC - Rutherford Appleton Laboratory
    Harwell Oxford,
    Didcot
    OX11 0QX
    +44 1235 446326 
     
    
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/