[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] transform syntax to build complex ads possible?



Hi,

still struggling with the $STRING but I feel like making progress. 

Maybe you have some helpful comments for my testing below... 

So I was using the test transform

JOB_TRANSFORM_MYTEST @=END
  Y = 123
  tmp = [a="Project"]["a"]
  SET Z1 = tmp
  SET Z2 = $STRING(tmp)
  SET Z3 = "tmp"
@END

Then 

# condor_config_val -verbose JOB_TRANSFORM_MYTEST
JOB_TRANSFORM_MYTEST = Y = 123
tmp = [a="Project"]["a"]
SET Z1 = tmp
SET Z2 = tmp
SET Z3 = "tmp"
 # at: /etc/condor/config.d/99debug.conf, line 15
 # raw: JOB_TRANSFORM_MYTEST = Y = 123
tmp = [a="Project"]["a"]
SET Z1 = tmp
SET Z2 = $STRING(tmp)
SET Z3 = "tmp"

As far as I understand, the variable JOB_TRANSFORM_MYTEST is, due to its heredoc-like definition, at this stage just a multi-line string, where only the $*() functions are expanded?

But while Z1 and Z2 look the same in evaluated (non-raw), there is still a difference because when I submit a test job:

# condor_q -l
...
Z1 = tmp
Z2 = Project
Z3 = "tmp"

Here the scheduler should have used the multiline string as transform rules for the job ad. 

Testing with condor_transform_ads shows me that maybe the scheduler works with the raw value?

# cat test.ad 
Project = "test"
X = 1
# cat rules
Y = 123
tmp = [a="Project"]["a"]
SET Z1 = tmp
SET Z2 = $STRING(tmp)
SET Z3 = "tmp"
# condor_transform_ads -in test.ad -rules rules -out:json out.ad
# cat out.ad 
[
{
  "Project": "test",
  "X": 1,
  "Z1": "\/Expr(tmp)\/",
  "Z2": "\/Expr(Project)\/",
  "Z3": "tmp"
}
]

Not really sure why the tmp in Z1 stays unchanged while the one in Z2 is.

Is it a feature of $STRING? Something like...
 
expand $STRING(tmp) = eval tmp as expression + treat resulting value as string + convert string into an expression?

Still feels like I am missing some crucial insight into the parsing process...

Best
  Kruno

----- Original Message -----
> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
> Sent: Monday, 11 October, 2021 18:40:55
> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?

> EVALMACRO and $STRING() are both ways to evaluate-and-print
> 
> $STRING(AA)   will lookup AA, and evaluate the contents, then print the result.
> This has the side effect of removing quotes from things that evaluate to string
> values
> 
> EVALMACRO does the same, except there is no lookup, the thing that it evaluates
> and prints
> is the right hand side of the statement after it.
> 
> The big difference here is that $STRING() requires you to create a temp variable
> to hold what you want to evaluate, and it lets you specify a print format
> specifier for the printing part of the operation,. EVALMACRO is not limited to
> string evaluation, but it does not let you control the formatting of the print
> operation.
> 
> So yes, this is a bit confusing.  The thing to keep in mind is that there is not
> so much contexts as phases.
> $() substitution happens first,  it is basically text manipulation, except for a
> few $FUNC() substitution functions which will evaluate and print.  the rest of
> the $() substition is just a print operation.   It is not necessary that each
> little bit that is $() substituted be a valid expression, it can be a part of
> an expression.   only the final result after all of the $()
> subtitution/printing is done must be a valid classad expression.
> 
> -tj
> ________________________________
> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever,
> Krunoslav <krunoslav.sever@xxxxxxx>
> Sent: Saturday, October 9, 2021 3:11 PM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
> 
> Hmm,
> 
> it appears this remains very fiddly business that requires extreme care...
> 
> So I can confirm that your workarounds (EVALMACRO+SET and tmp+SET $STRING(tmp))
> work.
> 
> With the EVALMACRO+SET I can sort of see why, the EVALMACRO probably giving
> slightly broader access to the scope.
> 
> In that regard, I forgot that Project is not actually in the submit file itself,
> it is created by a previous transform, so it may be treated differently in this
> context.
> 
> With the second one, I am still struggling, specifically what the $STRING
> function brings to the table...
> 
> Also, I tried the EVALMACRO with my if chain and failed to build actual string
> values, i.e.
> 
> EVALMACRO tmp = [ x01 = ifThenElse("Project" =?= undefined, "BIRD_noop",
> undefined) ]["x01"]
> 
> sort of works but how I do get the evalmacro to treat the "BIRD_noop" as actual
> string? Probably easy, but didn't find it...
> 
> I guess I could put the string value into another tmp variable and then access
> that, but this defeats the purpose of simplification.
> 
> While testing around, I realized I can use the userMap function to drastically
> reduce our if-chain from 27 conditions to about 5 which may be sufficiently
> simple for our purposes.
> 
> But I am still interested in understanding the new(er) classad features as of
> 9.0.x - it seems they still have a lot of quirks. All of the following contexts
> appear to throw in a another wrinkle:
> 
> * old style -> implicitly transformed to new style
> * new style
> * config context
>  -> $() evaluated differently?
> * job transform context
>  -> from docs "superset of an earlier transform language based on New ClassAds"
> * router transform context
> 
> It's hard to keep all those apart - there is a lot in the docs but I guess it is
> a little too much and currently is still more confusing than helping...
> 
> Do you have some sort of cheat sheet summarizing the differences per context?
> That would be very helpful to me and probably others, too.
> 
> Best
>  Kruno
> 
> ----- Original Message -----
>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>> Sent: Friday, 8 October, 2021 22:16:16
>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
> 
>> I would need to see your actual transform.  this EVALSET statement
>>
>> EVALSET AccountingGroup = [
>>  x1 = ifThenElse(cond1, value1,undefined);
>>  x2 = x1 ?: ifThenElse(cond2, value2, undefined);
>>  ...
>>  x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default);
>> ]["x<n>"]
>>
>> is not allowed to span multiple lines.   so the net effect will be
>>
>> EVALSET AccountingGroup = [
>>
>> followed by some statements that do nothing
>>
>> For your simpler example.
>>
>> EVALSET TEST = [ a=Project ]["a"]
>>
>> you say Project is set in submit. Does that mean the job has an attribute called
>> Project?
>> Because this is an EVALSET,  it would set TEST to the whatever Project evaluates
>> to.
>>
>> If that is ending up being undefined,  I think the problem here he one of scope.
>> since Project is not defined in the temporary ad [a=Project] it ends up being
>> undefined when evaluated.
>>
>> you can work around this limitation by generating a textual representation  of
>> the expression you want,and then then using SET to change the job.  This works
>> because SET will parse the text as an expression.  There are two ways to
>> accomplish this
>>
>>  EVALMACRO tmp = [a="Project"]["a"]
>>  SET test = $(tmp)
>>
>> or
>>
>>   tmp = [a="Project"]["a"]
>>   SET test = $STRING(tmp)
>>
>> note the use of a="Project" rather than a=Project here
>>
>> -tj
>>
>> ________________________________
>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever,
>> Krunoslav <krunoslav.sever@xxxxxxx>
>> Sent: Friday, October 8, 2021 7:01 AM
>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>>
>> Hi,
>>
>> this looks like a useful feature and I tried to use it to break up our
>> IfThenElse() construct in our JobTransform:
>>
>> eval_set_AccountingGroup = ifThenElse( cond1, \
>>  value1 \
>>  ifThenElse(cond2, \
>>    value2, \
>>    ...
>>  ) \
>> )
>>
>> As condor_status -limit 1 -af '[a=Name]["a"]' resolves to the Name from the
>> machine ad, I tried the replacement
>>
>> EVALSET AccountingGroup = [
>>  x1 = ifThenElse(cond1, value1,undefined);
>>  x2 = x1 ?: ifThenElse(cond2, value2, undefined);
>>  ...
>>  x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default);
>> ]["x<n>"]
>>
>> A reconfig accepted the transform (judging from the D_FULLDEBUG output in
>> SchedLog) but the conditions failed to access job ad to be transformed.
>>
>> Testing this with a simpler transform like
>>
>> EVALSET TEST = [ a=Project ]["a"]
>>
>> set undefined in the transformed job, even though Project is set in the original
>> submit.
>>
>> Aa an aside, I initially tried with eval_set, where it also reordered the x<i>
>> lines which does not happen with EVALSET.
>>
>> I assume this is a bug? If so, I'd appreciate a fix somewhere down the line,
>> modifying the current construct is very error-prone...
>>
>> Maybe it is not even too hard to add a function like
>>
>>  ifThenElseChain(cond1, val1, ..., condn, valn, default)
>>
>> This would take care of short-circuiting which my replacement lacks - not too
>> important, but could be nice...
>>
>> Best
>>  Kruno
>>
>> ----- Original Message -----
>>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx>
>>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>>> Sent: Thursday, 7 October, 2021 18:03:28
>>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible?
>>
>>> the GROUP_SORT_EXPR cannot be a classad, because it needs to evaluate to an
>>> integer
>>> but the expression can contain one or more classads as part of the expression
>>>
>>> for example
>>>
>>>> condor_status -limit 1 -af '[a=1;b=2]["a"]'
>>> 1
>>>> condor_status -limit 1 -af '[a=1;b=2]["b"]'
>>> 2
>>>> condor_status -limit 1 -af '[a=1;b=2]["c"]'
>>> error
>>>
>>> In the above [a=1;b=2] declares a classad,  and ["a"]  does a lookup within that
>>> classad for attribute a.
>>> the result of evaluation will be 1
>>>
>>> -tj
>>> ________________________________
>>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Thomas
>>> Hartmann <thomas.hartmann@xxxxxxx>
>>> Sent: Thursday, October 7, 2021 8:32 AM
>>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>>> Subject: [HTCondor-users] transform syntax to build complex ads possible?
>>>
>>> Hi all,
>>>
>>> is it actually possible, to write a a class ad in the new transform
>>> syntax (i.e., w/o square brackets)?
>>>
>>> AFAIS it is no macro/function/... where I can return a value, or?
>>>
>>> Background is, that I would like to refactor a `GROUP_SORT_EXPR`, that
>>> has grown over time into a forest of IfThenElse()'s.
>>>
>>> If I get it correctly, a 'macro' in the []-syntax is effectively a
>>> complex class ad. But I am not sure, if I can evaluate/set a new
>>> transform-like expression directly into an ad?
>>>
>>> Cheers,
>>>   Thomas
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
>> --
>> ------------------------------------------------------------------------
>> Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
>>                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
>>                                                            Notkestr. 85
>> phone:  +49-40-8998-1648                                   22607 Hamburg
>> e-mail: krunoslav.sever@xxxxxxx                                  Germany
>> ------------------------------------------------------------------------
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> --
> ------------------------------------------------------------------------
> Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
>                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
>                                                            Notkestr. 85
> phone:  +49-40-8998-1648                                   22607 Hamburg
> e-mail: krunoslav.sever@xxxxxxx                                  Germany
> ------------------------------------------------------------------------
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

-- 
------------------------------------------------------------------------
Krunoslav Sever            Deutsches Elektronen-Synchrotron (IT-Systems)
                        Ein Forschungszentrum der Helmholtz-Gemeinschaft
                                                            Notkestr. 85
phone:  +49-40-8998-1648                                   22607 Hamburg
e-mail: krunoslav.sever@xxxxxxx                                  Germany
------------------------------------------------------------------------