[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Submit requirements & ignoring significant attributes
- Date: Thu, 27 Oct 2016 15:04:27 +0000
- From: John M Knoeller <johnkn@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] Submit requirements & ignoring significant attributes
REMOVE_SIGNIFICANT_ATTRIBUTES should work, provided that it is set before the SCHEDD was started (actually before
The first time the SCHEDD sees a job that has QDate in its requirements expression.)
The SCHEDD has an internal state variable that is a lifetime union of all significant attributes, setting REMOVE_SIGNIFICATION_ATTRIBUTES will keep attributes from being ADDED to this variable, but it will not remove them from this variable.
Are you saying QDate was still a significant attribute even after a restart?
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Fischer, Max (SCC)
Sent: Thursday, October 27, 2016 9:29 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Submit requirements & ignoring significant attributes
I'm just going to reply here as I burdened Frank with some of that code. :D
1. We do not just need to transform the job, but also track it over its entire lifetime. There are actually hooks doing most of the work. Having plain router style transforms is also not sufficient, as we must communicate with an external service. We could probably do without the update hooks, but the finalize hook is important for us: it is used to figure out whether the routing gave significant performance improvement.
2. Having a separate "cleanup/delay" route to delay and release unrouted jobs could work. The `( ( CurrentTime - QDate ) > ( 10 * 3 ) ) ` expression is only required to release jobs which are *not* routed by the specific route in question.
Number 2. is IMO the cleanest solution, but shouldn't REMOVE_SIGNIFICANT_ATTRIBUTES at least work? It seems to have no effect.
> Am 27.10.2016 um 15:18 schrieb Todd Tannenbaum <tannenba@xxxxxxxxxxx>:
> On 10/25/2016 10:34 AM, Frank Fischer wrote:
>> Hi all,
>> I'm facing some strange issues in my configuration file regarding
>> auto-clustering/significant attributes and adding requirements upon
>> job submission (in our setup we have a job route defined - we append
>> requirements in order to make sure the router acts before the jobs
>>> APPEND_REQUIREMENTS = ( (INPUT_FILES =?= UNDEFINED) || (HPDA_Route
>>> TRUE) || (( CurrentTime - QDate ) > ( $(JOB_ROUTER_POLLING_PERIOD) *
>>> )) )
> So the real problem you are trying to solve is to prevent a job from running before the job_router acted on it.
> I suggest you get rid of all your customization of SIGNIFICANT_ATTRIBUTES (you will need to do "condor_restart -fast" after this, a condor_reconfig is not enough), and get rid of references to CurrentTime and QDate in APPEND_REQUIREMENTS, and instead solve the real problem via one of two possibilities:
> 1. If you can run HTCondor v8.5 on your submit machine, perhaps you
> can use a "job transform" instead of using the job_router. A "job
> transform" allows the admin to edit an incoming job classad, and is
> performed by the schedd BEFORE the job enters the queue. See the
> manual at https://is.gd/5p3Wks or the ticket about this at
> 2. If you want to do this via the job_router, you could add to your
> config something like JobWasRouted = False SUBMIT_ATTRS =
> $(SUBMIT_ATTRS) JobWasRouted APPEND_REQUIREMENTS = ( JobWasRouted =?=
> True ) and then in your route rule include
> set_JobWasRouted = True;
> This way you achieve your goal without referencing QDate in your Requirements expression which, as you discovered, wreaks havoc with autoclustering.
> Hope the above helps,
>> The following configuration values (should) handle significant attributes:
>>> # Round attributes (up) for better AutoClustering.
>>> # 25%: 112 => 125, 1133 => 1250
>>> # 3 : 112 => 1000, 1212 => 2000
>>> SCHEDD_ROUND_ATTR_RequestWalltime = 3
>>> SCHEDD_ROUND_ATTR_RequestMemory = 20% SCHEDD_ROUND_ATTR_RequestDisk
>>> = 25%
>>> SIGNIFICANT_ATTRIBUTES = JobUniverse,WantDocker,\
>>> REMOVE_SIGNIFICANT_ATTRIBUTES = DiskUsage, QDate
>> So far so good.
>> Now here's an example of condor_q -autocluster -long:
>>> ServerTime = 1477408979
>>> AutoClusterId = 2836
>>> JobCount = 7
>>> Requirements = ( ( TARGET.CLOUDSITE == "BWFORCLUSTER" ) ) && ( ( (
>>> INPUT_FILES =?= undefined ) || ( HPDA_Route =?= true ) || ( (
>>> CurrentTime - QDate ) > ( 10 * 3 ) ) ) ) && ( TARGET.Arch == "X86_64"
>>> ) && ( TARGET.OpSys == "LINUX" ) && ( TARGET.Disk >= RequestDisk )
>>> && ( TARGET.Memory >= RequestMemory ) && ( TARGET.HasFileTransfer )
>>> RequestDisk = DiskUsage JobIds = "175098.1 ... 175098.16"
>>> JobUniverse = 5
>>> RequestCpus = 1
>>> DiskUsage = 750000
>>> RequestWalltime = 69000
>>> QDate = 1477393245
>>> RemoteJob = true
>>> RequestMemory = 4000
>> Apparently QDate and DiskUsage are NOT removed from significant
>> attributes, although I explicitly told HTCondor to do so.
>> Am I missing something or do you see an error in my though process?
>> I'm out of ideas, what I could try to increase the number of
>> clustered jobs.
>> Thanks & regards
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
>> with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> The archives can be found at:
> Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
> Center for High Throughput Computing Department of Computer Sciences
> HTCondor Technical Lead 1210 W. Dayton St. Rm #4257
> Phone: (608) 263-7132 Madison, WI 53706-1685
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> The archives can be found at: