Re: [HTCondor-users] chaining CondorCE routes?

[Brian Lin <blin@xxxxxxxxxxx>]

Hi Thomas,

As long as you're using HTCondor 8.9.7+, you could theoretically use the 
new job transform syntax in the job router. However, the core of 
HTCondor-CE won't support the new syntax until 5.1.0.

If you're feeling dangerously adventurous, I've got a partially 
reviewed, untested pull request in to add this support: 
https://github.com/htcondor/htcondor-ce/pull/435

- Brian

On 3/25/21 10:23 AM, Thomas Hartmann wrote:
> Hi Brian,
>
> ah, OK - many thanks for the clarification!
>
> I will better move the common parts to the JOB_ROUTER_DEFAULT (the 
> native transform syntax is not yet supported here for 4.4.1, or? I 
> noticed in the meantime, that I had forgot another ';' again...)
>
> The 'JOB_ROUTE_NAMES' was probably a copy&paste error, when I copied 
> it from my previous 'JOB_ROUTER_ENTRIES' line :-/
>
> Cheers and thanks,
> Â Thomas
>
>
>
> On 25/03/2021 15.36, Brian Lin wrote:
> > Hi Thomas,
> >
> > Unfortunately, routes are exclusive and terminal so your job will 
> > only apply the rules in ROUTE_TEST_1 unless you have an earlier route 
> > that matches in JOB_ROUTE_NAMES (did you mean JOB_ROUTER_ROUTE_NAMES 
> > <https://htcondor.readthedocs.io/en/latest/admin-manual/configuration-macros.html#JOB_ROUTER_ROUTE_NAMES>?). 
> > This is easier to do in the new job router transform syntax (we're 
> > aiming to provide this in 5.1.0): there will be pre-route transforms, 
> > post-route transforms, and route transforms each with their own 
> > REQUIREMENTS expression to determine whether or not a transform is 
> > applied to a given job. So in this system, you could do something 
> > like [1] to achieve what you want.
> >
> > In the meantime, you can either duplicate the rules across routes or 
> > use the 'set_' or 'eval_set_' directives to add ClassAd expressions 
> > to JOB_ROUTER_DEFAULTS that are conditional upon route 1 + route 2 
> > requirements 
> > (https://htcondor-ce.readthedocs.io/en/latest/batch-system-integration/#setting-attributes-for-all-routes).
> >
> > - Brian
> >
> > [1]
> >
> > JOB_TRANSFORM_ROUTE_TEST_1 @=end
> > ÂÂ REQUIREMENTS True
> > ÂÂ SET GOTROUTE1 True
> > @end
> >
> > JOB_TRANSFORM_ROUTE_TEST_2 @=end
> > ÂÂ REQUIREMENTS True
> > ÂÂ SET GOTROUTE2 True
> > @end
> >
> > JOB_ROUTER_PRE_ROUTE_TRANSFORMS = $(JOB_ROUTER_PRE_ROUTE_TRANSFORMS) 
> > ROUTE_TEST_1 ROUTE_TEST_2
> >
> > JOB_ROUTER_ROUTE_Local_Condor @=jrt
> > ÂÂ TargetUniverse = 5
> > @jrt
> >
> > JOB_ROUTER_ROUTE_NAMES = Local_Condor
> >
> > On 3/25/21 5:59 AM, Thomas Hartmann wrote:
> > > Hi all,
> > >
> > > is it actually possible to chain routes and apply multiple to a 
> > > matching job?
> > >
> > > I have been trying something like [1], where I define multiple 
> > > routes and which should apply to all jobs (~`Requirements = True`)Â 
> > > -Â with the intention to modularize rules a bit. But I have not 
> > > managed to convince the CE to apply all matching(?) routes to a job.
> > > Thing is, that I have also tried to also order the routes, but it 
> > > seems, that while all the routes are added to the routing table [3] 
> > > only the first one gets applied [2].
> > >
> > > Versions in use are CondorCE 4.4.1 with Condor 8.9.11 [4]
> > >
> > > Cheers and thanks for ideas,
> > > Â Thomas
> > >
> > > [1]
> > > > cat 99_10_routedebug.conf
> > >
> > > ROUTETEST1 @=end
> > > [
> > > Â TargetUniverse = 5;
> > > Â name = "ROUTE_TEST_1";
> > > Â Requirements = True;
> > > Â set_GOTROUTE1=True
> > > ]
> > > @end
> > >
> > > ROUTETEST2 @=end
> > > [
> > > Â TargetUniverse = 5;
> > > Â name = "ROUTE_TEST_2";
> > > Â Requirements = True;
> > > Â set_GOTROUTE2=True
> > > ]
> > > @end
> > >
> > > JOB_ROUTER_ENTRIES = $(JOB_ROUTER_ENTRIES) $(ROUTETEST1) $(ROUTETEST2)
> > > JOB_ROUTE_NAMES = $(JOB_ROUTE_NAMES) $(ROUTETEST1) $(ROUTETEST2)
> > >
> > >
> > > [2]
> > > 03/25/21 11:48:08 JobRouter: Checking for candidate jobs. routing 
> > > table is:
> > > Route NameÂÂÂÂÂÂÂÂÂÂÂÂÂÂ SourceÂÂÂÂÂ Submitted/Max Idle/Max Throttle 
> > > Recent: Started Succeeded Failed
> > > ROUTE_TEST_1ÂÂÂÂÂÂÂÂÂÂÂÂ entries:1ÂÂÂÂÂÂÂÂ 2/Â 10000ÂÂÂÂÂÂ 2/ 2000 
> > > noneÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1ÂÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0
> > > ROUTE_TEST_2ÂÂÂÂÂÂÂÂÂÂÂÂ entries:2ÂÂÂÂÂÂÂÂ 0/Â 10000ÂÂÂÂÂÂ 0/ 2000 
> > > noneÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0ÂÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0
> > > Local_CondorÂÂÂÂÂÂÂÂÂÂÂÂ entries:0ÂÂÂÂÂÂÂÂ 2/Â 10000ÂÂÂÂÂÂ 2/ 2000 
> > > noneÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0ÂÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0
> > >
> > >
> > > [3]
> > > root@grid-htcondorce-dev: [/etc/condor-ce/config.d] 
> > > condor_ce_config_val JOB_ROUTE_NAMES JOB_ROUTER_ENTRIES
> > > Â[
> > > TargetUniverse = 5;
> > > name = "ROUTE_TEST_1";
> > > Requirements = True;
> > > set_GOTROUTE1=True
> > > ] [
> > > TargetUniverse = 5;
> > > name = "ROUTE_TEST_2";
> > > Requirements = True;
> > > set_GOTROUTE2=True
> > > ]
> > >
> > >
> > > [
> > > TargetUniverse = 5;
> > > name = "Local_Condor";
> > > ] [
> > > TargetUniverse = 5;
> > > name = "ROUTE_TEST_1";
> > > Requirements = True;
> > > set_GOTROUTE1=True
> > > ] [
> > > TargetUniverse = 5;
> > > name = "ROUTE_TEST_2";
> > > Requirements = True;
> > > set_GOTROUTE2=True
> > > ]
> > >
> > >
> > >
> > > [4]
> > > condor-8.9.11-1.el7.x86_64
> > > condor-boinc-7.16.11-1.el7.x86_64
> > > condor-classads-8.9.11-1.el7.x86_64
> > > condor-externals-8.9.11-1.el7.x86_64
> > > condor-procd-8.9.11-1.el7.x86_64
> > > htcondor-ce-4.4.1-3.el7.noarch
> > > htcondor-ce-apel-4.4.1-3.el7.noarch
> > > htcondor-ce-bdii-4.4.1-3.el7.noarch
> > > htcondor-ce-client-4.4.1-3.el7.noarch
> > > htcondor-ce-condor-4.4.1-3.el7.noarch
> > > htcondor-ce-view-4.4.1-3.el7.noarch
> > > python2-condor-8.9.11-1.el7.x86_64
> > > python3-condor-8.9.11-1.el7.x86_64
> > >
> > >
> > > _______________________________________________
> > > HTCondor-users mailing list
> > > To unsubscribe, send a message tohtcondor-users-request@xxxxxxxxxxx 
> > > with a
> > > subject: Unsubscribe
> > > You can also unsubscribe by visiting
> > > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> > >
> > > The archives can be found at:
> > > https://lists.cs.wisc.edu/archive/htcondor-users/