[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] transform syntax to build complex ads possible?



EVALMACRO and $STRING() are both ways to evaluate-and-print 

$STRING(AA)   will lookup AA, and evaluate the contents, then print the result.
This has the side effect of removing quotes from things that evaluate to string values

EVALMACRO does the same, except there is no lookup, the thing that it evaluates and prints
is the right hand side of the statement after it.

The big difference here is that $STRING() requires you to create a temp variable to hold what you want to evaluate, and it lets you specify a print format specifier for the printing part of the operation,. EVALMACRO is not limited to string evaluation, but it does not let you control the formatting of the print operation. 

So yes, this is a bit confusing.  The thing to keep in mind is that there is not so much contexts as phases. 
$() substitution happens first,  it is basically text manipulation, except for a few $FUNC() substitution functions which will evaluate and print.  the rest of the $() substition is just a print operation.   It is not necessary that each little bit that is $() substituted be a valid _expression_, it can be a part of an _expression_.   only the final result after all of the $() subtitution/printing is done must be a valid classad _expression_.

-tj

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever, Krunoslav <krunoslav.sever@xxxxxxx>
Sent: Saturday, October 9, 2021 3:11 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
 
Hmm,

it appears this remains very fiddly business that requires extreme care...

So I can confirm that your workarounds (EVALMACRO+SET and tmp+SET $STRING(tmp)) work.

With the EVALMACRO+SET I can sort of see why, the EVALMACRO probably giving slightly broader access to the scope.

In that regard, I forgot that Project is not actually in the submit file itself, it is created by a previous transform, so it may be treated differently in this context.

With the second one, I am still struggling, specifically what the $STRING function brings to the table...

Also, I tried the EVALMACRO with my if chain and failed to build actual string values, i.e.

EVALMACRO tmp = [ x01 = ifThenElse("Project" =?= undefined, "BIRD_noop", undefined) ]["x01"]

sort of works but how I do get the evalmacro to treat the "BIRD_noop" as actual string? Probably easy, but didn't find it...

I guess I could put the string value into another tmp variable and then access that, but this defeats the purpose of simplification.

While testing around, I realized I can use the userMap function to drastically reduce our if-chain from 27 conditions to about 5 which may be sufficiently simple for our purposes.

But I am still interested in understanding the new(er) classad features as of 9.0.x - it seems they still have a lot of quirks. All of the following contexts appear to throw in a another wrinkle:

* old style -> implicitly transformed to new style
* new style
* config context
  -> $() evaluated differently?
* job transform context
  -> from docs "superset of an earlier transform language based on New ClassAds"
* router transform context

It's hard to keep all those apart - there is a lot in the docs but I guess it is a little too much and currently is still more confusing than helping...

Do you have some sort of cheat sheet summarizing the differences per context? That would be very helpful to me and probably others, too.

Best
  Kruno

----- Original Message -----
> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
> Sent: Friday, 8 October, 2021 22:16:16
> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?

> I would need to see your actual transform.  this EVALSET statement
>
> EVALSET AccountingGroup = [
>  x1 = ifThenElse(cond1, value1,undefined);
>  x2 = x1 ?: ifThenElse(cond2, value2, undefined);
>  ...
>  x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default);
> ]["x<n>"]
>
> is not allowed to span multiple lines.   so the net effect will be
>
> EVALSET AccountingGroup = [
>
> followed by some statements that do nothing
>
> For your simpler example.
>
> EVALSET TEST = [ a=Project ]["a"]
>
> you say Project is set in submit. Does that mean the job has an attribute called
> Project?
> Because this is an EVALSET,  it would set TEST to the whatever Project evaluates
> to.
>
> If that is ending up being undefined,  I think the problem here he one of scope.
> since Project is not defined in the temporary ad [a=Project] it ends up being
> undefined when evaluated.
>
> you can work around this limitation by generating a textual representation  of
> the _expression_ you want,and then then using SET to change the job.  This works
> because SET will parse the text as an _expression_.  There are two ways to
> accomplish this
>
>  EVALMACRO tmp = [a="Project"]["a"]
>  SET test = $(tmp)
>
> or
>
>   tmp = [a="Project"]["a"]
>   SET test = $STRING(tmp)
>
> note the use of a="Project" rather than a=Project here
>
> -tj
>
> ________________________________
> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever,
> Krunoslav <krunoslav.sever@xxxxxxx>
> Sent: Friday, October 8, 2021 7:01 AM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>
> Hi,
>
> this looks like a useful feature and I tried to use it to break up our
> IfThenElse() construct in our JobTransform:
>
> eval_set_AccountingGroup = ifThenElse( cond1, \
>  value1 \
>  ifThenElse(cond2, \
>    value2, \
>    ...
>  ) \
> )
>
> As condor_status -limit 1 -af '[a=Name]["a"]' resolves to the Name from the
> machine ad, I tried the replacement
>
> EVALSET AccountingGroup = [
>  x1 = ifThenElse(cond1, value1,undefined);
>  x2 = x1 ?: ifThenElse(cond2, value2, undefined);
>  ...
>  x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default);
> ]["x<n>"]
>
> A reconfig accepted the transform (judging from the D_FULLDEBUG output in
> SchedLog) but the conditions failed to access job ad to be transformed.
>
> Testing this with a simpler transform like
>
> EVALSET TEST = [ a=Project ]["a"]
>
> set undefined in the transformed job, even though Project is set in the original
> submit.
>
> Aa an aside, I initially tried with eval_set, where it also reordered the x<i>
> lines which does not happen with EVALSET.
>
> I assume this is a bug? If so, I'd appreciate a fix somewhere down the line,
> modifying the current construct is very error-prone...
>
> Maybe it is not even too hard to add a function like
>
>  ifThenElseChain(cond1, val1, ..., condn, valn, default)
>
> This would take care of short-circuiting which my replacement lacks - not too
> important, but could be nice...
>
> Best
>  Kruno
>
> ----- Original Message -----
>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>> Sent: Thursday, 7 October, 2021 18:03:28
>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>
>> the GROUP_SORT_EXPR cannot be a classad, because it needs to evaluate to an
>> integer
>> but the _expression_ can contain one or more classads as part of the _expression_
>>
>> for example
>>
>>> condor_status -limit 1 -af '[a=1;b=2]["a"]'
>> 1
>>> condor_status -limit 1 -af '[a=1;b=2]["b"]'
>> 2
>>> condor_status -limit 1 -af '[a=1;b=2]["c"]'
>> error
>>
>> In the above [a=1;b=2] declares a classad,  and ["a"]  does a lookup within that
>> classad for attribute a.
>> the result of evaluation will be 1
>>
>> -tj
>> ________________________________
>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Thomas
>> Hartmann <thomas.hartmann@xxxxxxx>
>> Sent: Thursday, October 7, 2021 8:32 AM
>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>> Subject: [HTCondor-users] transform syntax to build complex ads possible?
>>
>> Hi all,
>>
>> is it actually possible, to write a a class ad in the new transform
>> syntax (i.e., w/o square brackets)?
>>
>> AFAIS it is no macro/function/... where I can return a value, or?
>>
>> Background is, that I would like to refactor a `GROUP_SORT_EXPR`, that
>> has grown over time into a forest of IfThenElse()'s.
>>
>> If I get it correctly, a 'macro' in the []-syntax is effectively a
>> complex class ad. But I am not sure, if I can evaluate/set a new
>> transform-like _expression_ directly into an ad?
>>
>> Cheers,
>>   Thomas
>>
>>
>>
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>
> --
> ------------------------------------------------------------------------
> Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
>                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
>                                                            Notkestr. 85
> phone:  +49-40-8998-1648                                   22607 Hamburg
> e-mail: krunoslav.sever@xxxxxxx                                  Germany
> ------------------------------------------------------------------------
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

--
------------------------------------------------------------------------
Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
                                                            Notkestr. 85
phone:  +49-40-8998-1648                                   22607 Hamburg
e-mail: krunoslav.sever@xxxxxxx                                  Germany
------------------------------------------------------------------------
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/