[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] TARGET.Runtime >= RequestRuntime



Hi Ben,

jep, you're probably right and we will certainly use the 'unified' feature once it's there. You had more luck in choosing your personal faeture-name than us ;) 

Best
Chris 

-- 
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx

----- UrsprÃngliche Mail -----
Von: "Ben Jones" <ben.dylan.jones@xxxxxxxxx>
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
Gesendet: Dienstag, 5. November 2019 10:31:10
Betreff: Re: [HTCondor-users] TARGET.Runtime >= RequestRuntime

Hi Chris,

FWIW we use +MaxRuntime for your RequestRuntime. And funnily enough I had a user asking about the various different ways sites define this as they wanted to have their submit framework check for it. Can't help but think it would be useful if the concept was builtin...

cheers,
Ben

> On 5 Nov 2019, at 10:17, Beyer, Christoph <christoph.beyer@xxxxxxx> wrote:
> 
> Hi John,
> 
> we talked with Todd about this at the condor-week and indeed it would be easiest way to have a blacklist option that we could use to exclude 'requestruntime' from being built in the requirement string on the production pool.
> 
> We will then test the new condor 'runtime-option' as a possible replacement of our current solution in the test pool :) 
> 
> Best
> Chris
> 
> -- 
> Christoph Beyer
> DESY Hamburg
> IT-Department
> 
> Notkestr. 85
> Building 02b, Room 009
> 22607 Hamburg
> 
> phone:+49-(0)40-8998-2317
> mail: christoph.beyer@xxxxxxx
> 
> ----- UrsprÃngliche Mail -----
> Von: "johnkn" <johnkn@xxxxxxxxxxx>
> An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
> Gesendet: Montag, 29. Juli 2019 16:41:59
> Betreff: Re: [HTCondor-users] TARGET.Runtime >= RequestRuntime
> 
> For 8.9.2 and 8.9.3 your options are to rename the attribute, or to have your users add TARGET.Runtime to their submit file Requirements.
> 
> Something like this should do. 
> 
> Requirements = (TRUE || TARGET.Runtime isnât undefined)
> 
> the True ||  part will make the matchmaker prune that subclause before it does any matchmaking, and the mention of TARGET.Runtime
> in your submit file Requirements will disable the automatic addition of (TARGET.Runtime >= RequestRuntime).
> 
> Or you could  rename your attribute to  RuntimeRequest or perhaps QuotaRuntime.  As the code is currently written,  *anything* that starts with the word Request is treated as a custom resource,  even +Request_Runtime.  (you would end up with a custom resource called _runtime). 
> I think I should change this for 8.9.4 so at the very least it won't treat anything with an _ in then name as a valid resource name.
> 
> We have been discussing adding something like request_runtime to HTCondor,  so jobs can indicate their expected and/or maximum runtime.  
> So your choice of RequestRuntime is very likely to conflict with future HTCondor.   I would consider adding a blacklist for resource names to the submit code if you need us to.    So you could configure
> 
> SUBMIT_BLACKLIST_RESOURCE_NAMES = Runtime
> 
> And submit would then ignore +RequestRuntime.   
> 
> But as I say, this particular name is likely to conflict with future HTCondor work that we are currently discussing, so I would recommend that you just choose a different name for your attribute.
> 
> -tj
> 
> -----Original Message-----
> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Beyer, Christoph
> Sent: Monday, July 29, 2019 4:53 AM
> To: htcondor-users <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] TARGET.Runtime >= RequestRuntime
> 
> Hi John,
> 
> did not recognize this e-mail at first, thanks for getting back to us :) 
> 
> As you say the behaviour changed with 8.9.2 and a lokal option that we use is now triggering the 'extended' requirement-expression. 
> 
> We use "+RequestRuntime = <time in sec>" as a custom thing to classify the job and apply the quota surplus accordingly. 
> 
> As the workernodes do not have a default runtime the created requirement-expression is not able to be fullfilled. From your mail I assume I have the following options: 
> 
> - rename "+RequestRuntime ..." into something like "+MYRuntime ..." (rather unelegant as we did educate the user to use "+RequestRuntime ...") 
> - establish a workernode classadd with unlimited runtime
> - put the same runtime into the jobclassadd as is requested, would that be  "RequestRuntime" or "request_runtime" ?
> 
> Maybe there is a more elegant way to tinker the requirement expression and get rid of the 'reques_runtime' part ? 
> 
> Best
> Christoph
> 
> -- 
> Christoph Beyer
> DESY Hamburg
> IT-Department
> 
> Notkestr. 85
> Building 02b, Room 009
> 22607 Hamburg
> 
> phone:+49-(0)40-8998-2317
> mail: christoph.beyer@xxxxxxx
> 
> ----- UrsprÃngliche Mail -----
> Von: "johnkn" <johnkn@xxxxxxxxxxx>
> An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
> Gesendet: Dienstag, 23. Juli 2019 21:13:39
> Betreff: Re: [HTCondor-users] TARGET.Runtime >= RequestRuntime
> 
> In condor_submit,  any keyword that starts with request_  is presumed to be request for a custom resource.  
> Everthing after the _ is presumed to be the resource type name
> 
> The resource names  Cpus, Disk, and Memory get special treatment.
> The resource name Cpu is treated as an error, because that is a common mistake. 
> 
> In the examples below, assume Tag is the custom resource name.
> 
> so 
> request_tag = <expression>
> 
> causes
> 
> RequestTag = <expression>
> 
> to be added to the job ClassAd, and the Requirements expression will have 
> 
> (TARGET.Tag >= RequestTag)
> 
>  or 
> 
> regexp(RequestTag, TARGET.TAG)
> 
> added to it.  the second expression above is used when <expression > is a string literal.
> 
> Starting with 8.9.2.  The Requirements expression will be changed when you use 
> 
>  +RequestTag = <expression>
> 
> as well as when you use 
> 
>  request_tag = <expression>
> 
> in your submit file.
> 
> if the requirements already references RequestTag,  then it will not be modified.
> 
> 
> -tj
> 
> -----Original Message-----
> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Beyer, Christoph
> Sent: Friday, July 19, 2019 5:24 AM
> To: htcondor-users <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] TARGET.Runtime >= RequestRuntime
> 
> Hi,
> 
> any thoughts on this ? 
> 
> Best
> Christoph
> 
> -- 
> Christoph Beyer
> DESY Hamburg
> IT-Department
> 
> Notkestr. 85
> Building 02b, Room 009
> 22607 Hamburg
> 
> phone:+49-(0)40-8998-2317
> mail: christoph.beyer@xxxxxxx
> 
> ----- UrsprÃngliche Mail -----
> Von: "Christoph Beyer" <christoph.beyer@xxxxxxx>
> An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
> Gesendet: Mittwoch, 17. Juli 2019 14:48:18
> Betreff: TARGET.Runtime >= RequestRuntime
> 
> Hi Todd et al,
> 
> this is probably rather a question for the developer or at last those who are aware of the source code I am afraid. 
> 
> I do have some user using a python framework and by doing so they trigger the requirement 'TARGET.Runtime >= RequestRuntime' in the r jon requirement which results in stuck jobs as TARGET.Runtime is not set by default on our workernodes.
> 
> I had this issue before triggered by somone who by mistake did not put '+RequestRuntime = XXX' in his submit file but 'request_runtime = XXX' which obviously was easy to fix. 
> 
> In the current case though I have no idea what triggers 'TARGET.Runtime >= RequestRuntime' to show up in the job requirement - hopefully someone can enlighten me :) (?) 
> 
> Best
> Christoph
> 
> -- 
> Christoph Beyer
> DESY Hamburg
> IT-Department
> 
> Notkestr. 85
> Building 02b, Room 009
> 22607 Hamburg
> 
> phone:+49-(0)40-8998-2317
> mail: christoph.beyer@xxxxxxx
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/