[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] transform syntax to build complex ads possible?



Yes  $STRING(tmp) function will lookup tmp and if tmp has never been set in the variable table, it will use tmp as a literal.

-tj



From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever, Krunoslav <krunoslav.sever@xxxxxxx>
Sent: Saturday, October 16, 2021 7:08 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
 
Just noticed this: you actually gave the definition of $STRING(AA) in the mail before, that had totally slipped my mind.

 $STRING(AA)   will lookup AA, and evaluate the contents, then print the result.
 This has the side effect of removing quotes from things that evaluate to string values

So I might actually be spot-on with my idea on the $STRING during transform, if print = convert to _expression_.

But why does the condor_config_val then show

raw: SET Z2 = $STRING(tmp) -> expanded: SET Z2 = tmp

Lookup of tmp should result in undefined at that stage unless it defaults to "tmp" in that case?

Best
  Kruno

----- Original Message -----
> From: "Krunoslav Sever" <krunoslav.sever@xxxxxxx>
> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
> Sent: Saturday, 16 October, 2021 11:13:15
> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?

> Hi,
>
> still struggling with the $STRING but I feel like making progress.
>
> Maybe you have some helpful comments for my testing below...
>
> So I was using the test transform
>
> JOB_TRANSFORM_MYTEST @=END
>  Y = 123
>  tmp = [a="Project"]["a"]
>  SET Z1 = tmp
>  SET Z2 = $STRING(tmp)
>  SET Z3 = "tmp"
> @END
>
> Then
>
> # condor_config_val -verbose JOB_TRANSFORM_MYTEST
> JOB_TRANSFORM_MYTEST = Y = 123
> tmp = [a="Project"]["a"]
> SET Z1 = tmp
> SET Z2 = tmp
> SET Z3 = "tmp"
> # at: /etc/condor/config.d/99debug.conf, line 15
> # raw: JOB_TRANSFORM_MYTEST = Y = 123
> tmp = [a="Project"]["a"]
> SET Z1 = tmp
> SET Z2 = $STRING(tmp)
> SET Z3 = "tmp"
>
> As far as I understand, the variable JOB_TRANSFORM_MYTEST is, due to its
> heredoc-like definition, at this stage just a multi-line string, where only the
> $*() functions are expanded?
>
> But while Z1 and Z2 look the same in evaluated (non-raw), there is still a
> difference because when I submit a test job:
>
> # condor_q -l
> ...
> Z1 = tmp
> Z2 = Project
> Z3 = "tmp"
>
> Here the scheduler should have used the multiline string as transform rules for
> the job ad.
>
> Testing with condor_transform_ads shows me that maybe the scheduler works with
> the raw value?
>
> # cat test.ad
> Project = "test"
> X = 1
> # cat rules
> Y = 123
> tmp = [a="Project"]["a"]
> SET Z1 = tmp
> SET Z2 = $STRING(tmp)
> SET Z3 = "tmp"
> # condor_transform_ads -in test.ad -rules rules -out:json out.ad
> # cat out.ad
> [
> {
>  "Project": "test",
>  "X": 1,
>  "Z1": "\/Expr(tmp)\/",
>  "Z2": "\/Expr(Project)\/",
>  "Z3": "tmp"
> }
> ]
>
> Not really sure why the tmp in Z1 stays unchanged while the one in Z2 is.
>
> Is it a feature of $STRING? Something like...
>
> expand $STRING(tmp) = eval tmp as _expression_ + treat resulting value as string +
> convert string into an _expression_?
>
> Still feels like I am missing some crucial insight into the parsing process...
>
> Best
>  Kruno
>
> ----- Original Message -----
>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>> Sent: Monday, 11 October, 2021 18:40:55
>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>
>> EVALMACRO and $STRING() are both ways to evaluate-and-print
>>
>> $STRING(AA)   will lookup AA, and evaluate the contents, then print the result.
>> This has the side effect of removing quotes from things that evaluate to string
>> values
>>
>> EVALMACRO does the same, except there is no lookup, the thing that it evaluates
>> and prints
>> is the right hand side of the statement after it.
>>
>> The big difference here is that $STRING() requires you to create a temp variable
>> to hold what you want to evaluate, and it lets you specify a print format
>> specifier for the printing part of the operation,. EVALMACRO is not limited to
>> string evaluation, but it does not let you control the formatting of the print
>> operation.
>>
>> So yes, this is a bit confusing.  The thing to keep in mind is that there is not
>> so much contexts as phases.
>> $() substitution happens first,  it is basically text manipulation, except for a
>> few $FUNC() substitution functions which will evaluate and print.  the rest of
>> the $() substition is just a print operation.   It is not necessary that each
>> little bit that is $() substituted be a valid _expression_, it can be a part of
>> an _expression_.   only the final result after all of the $()
>> subtitution/printing is done must be a valid classad _expression_.
>>
>> -tj
>> ________________________________
>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever,
>> Krunoslav <krunoslav.sever@xxxxxxx>
>> Sent: Saturday, October 9, 2021 3:11 PM
>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>>
>> Hmm,
>>
>> it appears this remains very fiddly business that requires extreme care...
>>
>> So I can confirm that your workarounds (EVALMACRO+SET and tmp+SET $STRING(tmp))
>> work.
>>
>> With the EVALMACRO+SET I can sort of see why, the EVALMACRO probably giving
>> slightly broader access to the scope.
>>
>> In that regard, I forgot that Project is not actually in the submit file itself,
>> it is created by a previous transform, so it may be treated differently in this
>> context.
>>
>> With the second one, I am still struggling, specifically what the $STRING
>> function brings to the table...
>>
>> Also, I tried the EVALMACRO with my if chain and failed to build actual string
>> values, i.e.
>>
>> EVALMACRO tmp = [ x01 = ifThenElse("Project" =?= undefined, "BIRD_noop",
>> undefined) ]["x01"]
>>
>> sort of works but how I do get the evalmacro to treat the "BIRD_noop" as actual
>> string? Probably easy, but didn't find it...
>>
>> I guess I could put the string value into another tmp variable and then access
>> that, but this defeats the purpose of simplification.
>>
>> While testing around, I realized I can use the userMap function to drastically
>> reduce our if-chain from 27 conditions to about 5 which may be sufficiently
>> simple for our purposes.
>>
>> But I am still interested in understanding the new(er) classad features as of
>> 9.0.x - it seems they still have a lot of quirks. All of the following contexts
>> appear to throw in a another wrinkle:
>>
>> * old style -> implicitly transformed to new style
>> * new style
>> * config context
>>  -> $() evaluated differently?
>> * job transform context
>>  -> from docs "superset of an earlier transform language based on New ClassAds"
>> * router transform context
>>
>> It's hard to keep all those apart - there is a lot in the docs but I guess it is
>> a little too much and currently is still more confusing than helping...
>>
>> Do you have some sort of cheat sheet summarizing the differences per context?
>> That would be very helpful to me and probably others, too.
>>
>> Best
>>  Kruno
>>
>> ----- Original Message -----
>>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
>>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>>> Sent: Friday, 8 October, 2021 22:16:16
>>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>>
>>> I would need to see your actual transform.  this EVALSET statement
>>>
>>> EVALSET AccountingGroup = [
>>>  x1 = ifThenElse(cond1, value1,undefined);
>>>  x2 = x1 ?: ifThenElse(cond2, value2, undefined);
>>>  ...
>>>  x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default);
>>> ]["x<n>"]
>>>
>>> is not allowed to span multiple lines.   so the net effect will be
>>>
>>> EVALSET AccountingGroup = [
>>>
>>> followed by some statements that do nothing
>>>
>>> For your simpler example.
>>>
>>> EVALSET TEST = [ a=Project ]["a"]
>>>
>>> you say Project is set in submit. Does that mean the job has an attribute called
>>> Project?
>>> Because this is an EVALSET,  it would set TEST to the whatever Project evaluates
>>> to.
>>>
>>> If that is ending up being undefined,  I think the problem here he one of scope.
>>> since Project is not defined in the temporary ad [a=Project] it ends up being
>>> undefined when evaluated.
>>>
>>> you can work around this limitation by generating a textual representation  of
>>> the _expression_ you want,and then then using SET to change the job.  This works
>>> because SET will parse the text as an _expression_.  There are two ways to
>>> accomplish this
>>>
>>>  EVALMACRO tmp = [a="Project"]["a"]
>>>  SET test = $(tmp)
>>>
>>> or
>>>
>>>   tmp = [a="Project"]["a"]
>>>   SET test = $STRING(tmp)
>>>
>>> note the use of a="Project" rather than a=Project here
>>>
>>> -tj
>>>
>>> ________________________________
>>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever,
>>> Krunoslav <krunoslav.sever@xxxxxxx>
>>> Sent: Friday, October 8, 2021 7:01 AM
>>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>>>
>>> Hi,
>>>
>>> this looks like a useful feature and I tried to use it to break up our
>>> IfThenElse() construct in our JobTransform:
>>>
>>> eval_set_AccountingGroup = ifThenElse( cond1, \
>>>  value1 \
>>>  ifThenElse(cond2, \
>>>    value2, \
>>>    ...
>>>  ) \
>>> )
>>>
>>> As condor_status -limit 1 -af '[a=Name]["a"]' resolves to the Name from the
>>> machine ad, I tried the replacement
>>>
>>> EVALSET AccountingGroup = [
>>>  x1 = ifThenElse(cond1, value1,undefined);
>>>  x2 = x1 ?: ifThenElse(cond2, value2, undefined);
>>>  ...
>>>  x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default);
>>> ]["x<n>"]
>>>
>>> A reconfig accepted the transform (judging from the D_FULLDEBUG output in
>>> SchedLog) but the conditions failed to access job ad to be transformed.
>>>
>>> Testing this with a simpler transform like
>>>
>>> EVALSET TEST = [ a=Project ]["a"]
>>>
>>> set undefined in the transformed job, even though Project is set in the original
>>> submit.
>>>
>>> Aa an aside, I initially tried with eval_set, where it also reordered the x<i>
>>> lines which does not happen with EVALSET.
>>>
>>> I assume this is a bug? If so, I'd appreciate a fix somewhere down the line,
>>> modifying the current construct is very error-prone...
>>>
>>> Maybe it is not even too hard to add a function like
>>>
>>>  ifThenElseChain(cond1, val1, ..., condn, valn, default)
>>>
>>> This would take care of short-circuiting which my replacement lacks - not too
>>> important, but could be nice...
>>>
>>> Best
>>>  Kruno
>>>
>>> ----- Original Message -----
>>>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
>>>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>>>> Sent: Thursday, 7 October, 2021 18:03:28
>>>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>>>
>>>> the GROUP_SORT_EXPR cannot be a classad, because it needs to evaluate to an
>>>> integer
>>>> but the _expression_ can contain one or more classads as part of the _expression_
>>>>
>>>> for example
>>>>
>>>>> condor_status -limit 1 -af '[a=1;b=2]["a"]'
>>>> 1
>>>>> condor_status -limit 1 -af '[a=1;b=2]["b"]'
>>>> 2
>>>>> condor_status -limit 1 -af '[a=1;b=2]["c"]'
>>>> error
>>>>
>>>> In the above [a=1;b=2] declares a classad,  and ["a"]  does a lookup within that
>>>> classad for attribute a.
>>>> the result of evaluation will be 1
>>>>
>>>> -tj
>>>> ________________________________
>>>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Thomas
>>>> Hartmann <thomas.hartmann@xxxxxxx>
>>>> Sent: Thursday, October 7, 2021 8:32 AM
>>>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>>>> Subject: [HTCondor-users] transform syntax to build complex ads possible?
>>>>
>>>> Hi all,
>>>>
>>>> is it actually possible, to write a a class ad in the new transform
>>>> syntax (i.e., w/o square brackets)?
>>>>
>>>> AFAIS it is no macro/function/... where I can return a value, or?
>>>>
>>>> Background is, that I would like to refactor a `GROUP_SORT_EXPR`, that
>>>> has grown over time into a forest of IfThenElse()'s.
>>>>
>>>> If I get it correctly, a 'macro' in the []-syntax is effectively a
>>>> complex class ad. But I am not sure, if I can evaluate/set a new
>>>> transform-like _expression_ directly into an ad?
>>>>
>>>> Cheers,
>>>>   Thomas
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> HTCondor-users mailing list
>>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>>> subject: Unsubscribe
>>>> You can also unsubscribe by visiting
>>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>>
>>>> The archives can be found at:
>>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>>
>>> --
>>> ------------------------------------------------------------------------
>>> Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
>>>                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
>>>                                                            Notkestr. 85
>>> phone:  +49-40-8998-1648                                   22607 Hamburg
>>> e-mail: krunoslav.sever@xxxxxxx                                  Germany
>>> ------------------------------------------------------------------------
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>>
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
>> --
>> ------------------------------------------------------------------------
>> Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
>>                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
>>                                                            Notkestr. 85
>> phone:  +49-40-8998-1648                                   22607 Hamburg
>> e-mail: krunoslav.sever@xxxxxxx                                  Germany
>> ------------------------------------------------------------------------
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>
> --
> ------------------------------------------------------------------------
> Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
>                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
>                                                            Notkestr. 85
> phone:  +49-40-8998-1648                                   22607 Hamburg
> e-mail: krunoslav.sever@xxxxxxx                                  Germany
> ------------------------------------------------------------------------
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

--
------------------------------------------------------------------------
Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
                                                            Notkestr. 85
phone:  +49-40-8998-1648                                   22607 Hamburg
e-mail: krunoslav.sever@xxxxxxx                                  Germany
------------------------------------------------------------------------
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/