Mailing List Archives
Public Access
|
|
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif)
|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[HTCondor-users] HTCondor doesn't run a job after some time
- Date: Tue, 15 Jun 2021 18:38:42 +0300 (MSK)
- From: "Dmitry A. Golubkov" <dmitry.golubkov@xxxxxxxxxxxxxx>
- Subject: [HTCondor-users] HTCondor doesn't run a job after some time
Dear all,
After several successive runs, htcondor ended up in a strange state:
---
Name OpSys Arch State Activity LoadAv Mem ActvtyTime
slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LINUX X86_64 Unclaimed Idle 0.000 1800 0+00:54:38
slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LINUX X86_64 Unclaimed Idle 0.000 1800 0+00:54:39
slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LINUX X86_64 Unclaimed Idle 0.000 1800 0+00:54:39
Total Owner Claimed Unclaimed Matched Preempting Backfill Drain
X86_64/LINUX 3 0 0 3 0 0 0 0
Total 3 0 0 3 0 0 0 0
-- Schedd: parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx : <10.42.0.171:46693?... @ 06/15/21 15:24:40
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE HOLD TOTAL JOB_IDS
user20001 ID: 91 6/15 14:53 _ _ 5 _ 5 91.0-4
Total for query: 5 jobs; 0 completed, 0 removed, 5 idle, 0 running, 0 held, 0 suspended
Total for all users: 5 jobs; 0 completed, 0 removed, 5 idle, 0 running, 0 held, 0 suspended
----
As you see, I have three partitionable slots, absolutely free, one job in the idle state which can be started but nothing happens for a long time (a have waited for 30 minutes). In condor_q details I have found:
---
Slots
Step Matched Condition
----- -------- ---------
[2] 3 OpSys == "LINUX"
[5] 3 Arch == "X86_64"
[7] 3 DA__P7__RUNENV_PYTHON3 >= 13
[9] 3 DA__P7__CLUSTER_NODE == "True"
[11] 3 TARGET.Disk >= RequestDisk
[13] 3 TARGET.Memory >= RequestMemory
[15] 3 TARGET.FileSystemDomain == MY.FileSystemDomain
No successful match recorded.
Last failed match: Tue Jun 15 14:55:24 2021
Reason for last match failure: PREEMPTION_REQUIREMENTS == False
091.004: Run analysis summary ignoring user priority. Of 3 machines,
0 are rejected by your job's requirements
0 reject your job because of their own requirements
0 match and are already running your jobs
0 match but are serving other users
3 are able to run your job
---
But all previous jobs finished successfully. And I use dynamic slots to run the jobs. Any ideas?
Thanks in advance,
Dmitry.[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:41.855576482Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:41.856041431Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:44.979382156Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000094; send_time=0.000177; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:39381>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.014027439Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:42.873905242Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.015025461Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.019209868Z condor_collector[5699]: (Sending 3 ads in response to query)
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:42.874740324Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:43.924868961Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.019243727Z condor_collector[50]: QueryWorker: forked new worker with id 5699 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.019256654Z condor_collector[5699]: Query info: matched=3; skipped=0; query_time=0.001048; send_time=0.000876; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:34899>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.040900716Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.041938530Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.041972432Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.042307399Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000086; send_time=0.000153; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:43989>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:43.925622511Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:44.983613655Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:44.985335962Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:46.046591112Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:46.048399280Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:33:27.673959862Z condor_procd[120]: no methods have determined process 6225 to be in a monitored family
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:33:44.020400970Z condor_procd[120]: no methods have determined process 5965 to be in a monitored family
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:33:44.020418120Z condor_procd[120]: ...snapshot complete
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:33:27.674004971Z condor_procd[120]: ...snapshot complete
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:31.579371376Z condor_shared_port[121]: About to update statistics in shared_port daemon ad file at /var/lock/condor/shared_port_ad :
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:31.579409234Z condor_shared_port[121]: ForkedChildrenCurrent = 0#012ForkedChildrenPeak = 0#012MyAddress = "<10.42.0.184:41841?addrs=10.42.0.184-41841&alias=pseven-htcondorexecute-deploy-89c5b44cd-xc4f9.pseven-htcondor&noUDP>"#012RequestsBlocked = 1#012RequestsFailed = 1#012RequestsPendingCurrent = 0#012RequestsPendingPeak = 5#012RequestsSucceeded = 856#012SharedPortCommandSinfuls = "<10.42.0.184:41841?alias=pseven-htcondorexecute-deploy-89c5b44cd-xc4f9.pseven-htcondor>"
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:44.079787594Z condor_procd[120]: taking a snapshot...
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:44.079850282Z condor_procd[120]: ProcAPI: read 8 pid entries out of 71 total entries in /proc
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:44.079982180Z condor_procd[120]: ProcAPI: new boottime = 1621268011; old_boottime = 1621268011; /proc/stat boottime = 1621268011; /proc/uptime boottime = 1621268011
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:44.080999186Z condor_procd[120]: process 5965 (not in monitored family) has exited
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:44.081038445Z condor_procd[120]: no methods have determined process 5979 to be in a monitored family
[pod/pseven-htcondorexecute-deploy-89c5b44cd-xc4f9/htcondorexecute] 2021-06-15T15:34:44.081052139Z condor_procd[120]: ...snapshot complete
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:27.723940869Z condor_procd[120]: taking a snapshot...
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:27.724002752Z condor_procd[120]: ProcAPI: read 8 pid entries out of 71 total entries in /proc
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:27.724177884Z condor_procd[120]: ProcAPI: new boottime = 1621268011; old_boottime = 1621268011; /proc/stat boottime = 1621268011; /proc/uptime boottime = 1621268011
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:27.725270075Z condor_procd[120]: process 6225 (not in monitored family) has exited
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:27.725331900Z condor_procd[120]: no methods have determined process 6239 to be in a monitored family
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:27.725466698Z condor_procd[120]: ...snapshot complete
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:31.111473492Z condor_shared_port[121]: About to update statistics in shared_port daemon ad file at /var/lock/condor/shared_port_ad :
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:33:41.842811108Z condor_procd[120]: no methods have determined process 6085 to be in a monitored family
[pod/pseven-htcondorexecute-deploy-89c5b44cd-pm29c/htcondorexecute] 2021-06-15T15:34:31.111494700Z condor_shared_port[121]: ForkedChildrenCurrent = 0#012ForkedChildrenPeak = 0#012MyAddress = "<10.42.0.183:46111?addrs=10.42.0.183-46111&alias=pseven-htcondorexecute-deploy-89c5b44cd-pm29c.pseven-htcondor&noUDP>"#012RequestsBlocked = 0#012RequestsFailed = 0#012RequestsPendingCurrent = 0#012RequestsPendingPeak = 7#012RequestsSucceeded = 918#012SharedPortCommandSinfuls = "<10.42.0.183:46111?alias=pseven-htcondorexecute-deploy-89c5b44cd-pm29c.pseven-htcondor>"
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:33:41.842818761Z condor_procd[120]: ...snapshot complete
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:31.527459098Z condor_shared_port[121]: About to update statistics in shared_port daemon ad file at /var/lock/condor/shared_port_ad :
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:31.527481008Z condor_shared_port[121]: ForkedChildrenCurrent = 0#012ForkedChildrenPeak = 0#012MyAddress = "<10.42.0.185:38323?addrs=10.42.0.185-38323&alias=pseven-htcondorexecute-deploy-89c5b44cd-94jgp.pseven-htcondor&noUDP>"#012RequestsBlocked = 1#012RequestsFailed = 1#012RequestsPendingCurrent = 0#012RequestsPendingPeak = 5#012RequestsSucceeded = 907#012SharedPortCommandSinfuls = "<10.42.0.185:38323?alias=pseven-htcondorexecute-deploy-89c5b44cd-94jgp.pseven-htcondor>"
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:41.899446956Z condor_procd[120]: taking a snapshot...
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:41.899467969Z condor_procd[120]: ProcAPI: read 8 pid entries out of 71 total entries in /proc
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:41.899484118Z condor_procd[120]: ProcAPI: new boottime = 1621268010; old_boottime = 1621268011; /proc/stat boottime = 1621268011; /proc/uptime boottime = 1621268010
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:41.899920612Z condor_procd[120]: process 6085 (not in monitored family) has exited
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:41.900015978Z condor_procd[120]: no methods have determined process 6099 to be in a monitored family
[pod/pseven-htcondorexecute-deploy-89c5b44cd-94jgp/htcondorexecute] 2021-06-15T15:34:41.900022196Z condor_procd[120]: ...snapshot complete
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.937740925Z condor_negotiator[51]: ---------- Started Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.937763855Z condor_negotiator[51]: Phase 1: Obtaining ads from collector ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.937767372Z condor_negotiator[51]: Getting startd private ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.938257765Z condor_collector[50]: Got QUERY_STARTD_PVT_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.938813755Z condor_collector[5700]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.938830125Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5700 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.939062199Z condor_collector[5700]: Query info: matched=3; skipped=0; query_time=0.000246; send_time=0.000216; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:42413>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.939168845Z condor_negotiator[51]: Getting Scheduler, Submitter and Machine ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.940083072Z condor_collector[5701]: (Sending 5 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.940094776Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5701 ( max 4 active 2 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.940866532Z condor_collector[5701]: Query info: matched=5; skipped=9; query_time=0.000243; send_time=0.000761; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:39839>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.941538061Z condor_negotiator[51]: Sorting 5 ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.941654281Z condor_negotiator[51]: Got ads: 5 public and 3 private
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.941664567Z condor_negotiator[51]: Public ads include 0 submitter, 3 startd
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.941667279Z condor_negotiator[51]: Phase 2: Performing accounting ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.944352561Z condor_negotiator[51]: Phase 3: Sorting submitter ads by priority ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.944367898Z condor_negotiator[51]: Starting prefetch round; 0 potential prefetches to do.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.944371084Z condor_negotiator[51]: Prefetch summary: 0 attempted, 0 successful.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.944373428Z condor_negotiator[51]: Phase 4.1: Negotiating with schedds ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.944375747Z condor_negotiator[51]: negotiateWithGroup resources used submitterAds length 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:46.944377956Z condor_negotiator[51]: ---------- Finished Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.077215260Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.078235110Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.080149459Z condor_collector[5702]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.080366470Z condor_collector[50]: QueryWorker: forked new worker with id 5702 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.081278096Z condor_collector[5702]: Query info: matched=3; skipped=0; query_time=0.000929; send_time=0.000972; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:38147>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.100413116Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:47.104441503Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.101243924Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.101285045Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:47.101558375Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000088; send_time=0.000144; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:41137>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:47.105888015Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.134290499Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.137170175Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.137638179Z condor_collector[5703]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.138011431Z condor_collector[50]: QueryWorker: forked new worker with id 5703 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.138833918Z condor_collector[5703]: Query info: matched=3; skipped=0; query_time=0.001045; send_time=0.000942; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:39433>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.154010768Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:48.156773481Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.154475203Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.154694502Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:48.154708368Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000053; send_time=0.000072; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:34767>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:48.158108583Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.185572958Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.186708823Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.188584195Z condor_collector[5704]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.188723409Z condor_collector[50]: QueryWorker: forked new worker with id 5704 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.189710442Z condor_collector[5704]: Query info: matched=3; skipped=0; query_time=0.000948; send_time=0.000926; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:37765>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.210252392Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:49.213115100Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.210835981Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.210865079Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:49.211115104Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000069; send_time=0.000091; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:40007>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:49.214209921Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.224677997Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.224975769Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.225776158Z condor_collector[5705]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.225847650Z condor_collector[50]: QueryWorker: forked new worker with id 5705 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.226131524Z condor_collector[5705]: Query info: matched=3; skipped=0; query_time=0.000335; send_time=0.000317; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:35131>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.233426449Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:50.235013502Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.233756117Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.233770997Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:50.233905029Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000039; send_time=0.000042; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:39639>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:50.235816962Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.124161473Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.125351878Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.127443283Z condor_collector[5706]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.127767000Z condor_collector[50]: QueryWorker: forked new worker with id 5706 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.128840137Z condor_collector[5706]: Query info: matched=3; skipped=0; query_time=0.001145; send_time=0.001130; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:44125>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.153096431Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:51.156663953Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.153734175Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.154409165Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.154429244Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000070; send_time=0.000093; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:33357>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:51.159129767Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.542599896Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.543895219Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.545506643Z condor_collector[5707]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.545662992Z condor_collector[50]: QueryWorker: forked new worker with id 5707 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.546845698Z condor_collector[5707]: Query info: matched=3; skipped=0; query_time=0.000801; send_time=0.000614; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:44869>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.558975564Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:51.561233222Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.559495099Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.559522302Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:51.559637478Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000054; send_time=0.000076; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:37589>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:51.562033498Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.589062508Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.590248127Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.592076663Z condor_collector[5708]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.592275749Z condor_collector[50]: QueryWorker: forked new worker with id 5708 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.592967378Z condor_collector[5708]: Query info: matched=3; skipped=0; query_time=0.001042; send_time=0.000847; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:36961>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.618194913Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:52.623946924Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.619701592Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.619760975Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.619784242Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000180; send_time=0.000172; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:44993>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:52.625783469Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.951621807Z condor_negotiator[51]: ---------- Started Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.951680794Z condor_negotiator[51]: Phase 1: Obtaining ads from collector ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.951695673Z condor_negotiator[51]: Getting startd private ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.953804731Z condor_collector[50]: Got QUERY_STARTD_PVT_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.955761807Z condor_collector[5709]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.955811432Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5709 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.956889365Z condor_collector[5709]: Query info: matched=3; skipped=0; query_time=0.001044; send_time=0.000975; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:37401>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.957307702Z condor_negotiator[51]: Getting Scheduler, Submitter and Machine ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.961715499Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5710 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.961780431Z condor_collector[5710]: (Sending 5 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.964646803Z condor_collector[5710]: Query info: matched=5; skipped=9; query_time=0.001210; send_time=0.002790; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:34511>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.966869394Z condor_negotiator[51]: Sorting 5 ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.967270264Z condor_negotiator[51]: Got ads: 5 public and 3 private
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.967310172Z condor_negotiator[51]: Public ads include 0 submitter, 3 startd
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.967325077Z condor_negotiator[51]: Phase 2: Performing accounting ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.972736750Z condor_negotiator[51]: Phase 3: Sorting submitter ads by priority ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.972787866Z condor_negotiator[51]: Starting prefetch round; 0 potential prefetches to do.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.972803689Z condor_negotiator[51]: Prefetch summary: 0 attempted, 0 successful.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.972819522Z condor_negotiator[51]: Phase 4.1: Negotiating with schedds ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.972877781Z condor_negotiator[51]: negotiateWithGroup resources used submitterAds length 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:52.972904173Z condor_negotiator[51]: ---------- Finished Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.653151606Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.654316683Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.656264557Z condor_collector[5711]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.656546522Z condor_collector[50]: QueryWorker: forked new worker with id 5711 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.657377783Z condor_collector[5711]: Query info: matched=3; skipped=0; query_time=0.000911; send_time=0.000883; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:45881>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.681349395Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:53.686612724Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.682737529Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.682798141Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:53.683128547Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000127; send_time=0.000174; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:36797>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:53.688107307Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.715909487Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.717195361Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.718871976Z condor_collector[5712]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.719158568Z condor_collector[50]: QueryWorker: forked new worker with id 5712 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.719904448Z condor_collector[5712]: Query info: matched=3; skipped=0; query_time=0.000916; send_time=0.000926; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:44855>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.744225905Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:54.749620603Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.745896184Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.745944405Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:54.745966184Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000152; send_time=0.000199; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:40735>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:54.751150533Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.778982295Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.780147487Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.781954147Z condor_collector[5713]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.782500720Z condor_collector[50]: QueryWorker: forked new worker with id 5713 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.783020755Z condor_collector[5713]: Query info: matched=3; skipped=0; query_time=0.000945; send_time=0.000875; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:35981>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.801995185Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:55.804846910Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.802480170Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.802507373Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:55.802722398Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000057; send_time=0.000086; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:37633>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:55.805679017Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.834297063Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.837082723Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.839494759Z condor_collector[5714]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.839543346Z condor_collector[50]: QueryWorker: forked new worker with id 5714 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.839565484Z condor_collector[5714]: Query info: matched=3; skipped=0; query_time=0.001033; send_time=0.001002; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:44577>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.854433201Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:56.856655613Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.854805262Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.854829766Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:56.856163136Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000042; send_time=0.000070; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:39585>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:56.857575322Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.883355238Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.884524869Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.886488473Z condor_collector[5715]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.886777434Z condor_collector[50]: QueryWorker: forked new worker with id 5715 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.887511123Z condor_collector[5715]: Query info: matched=3; skipped=0; query_time=0.001074; send_time=0.000912; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:44615>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.912164494Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:57.919223167Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.913372707Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.913439209Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:57.913988544Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000132; send_time=0.000191; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:42127>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:57.920734963Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.949830528Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.951052871Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.954045825Z condor_collector[5716]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.954100282Z condor_collector[50]: QueryWorker: forked new worker with id 5716 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.954122591Z condor_collector[5716]: Query info: matched=3; skipped=0; query_time=0.001018; send_time=0.000886; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:38679>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.973777897Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:58.977728252Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.974601432Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.974640626Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.974913147Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000098; send_time=0.000129; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:40963>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:34:58.980364297Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.980361345Z condor_negotiator[51]: ---------- Started Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.980393506Z condor_negotiator[51]: Phase 1: Obtaining ads from collector ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.980405463Z condor_negotiator[51]: Getting startd private ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.980819479Z condor_collector[50]: Got QUERY_STARTD_PVT_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.981948455Z condor_collector[5717]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.981971925Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5717 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.982636447Z condor_negotiator[51]: Getting Scheduler, Submitter and Machine ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.984659234Z condor_collector[5717]: Query info: matched=3; skipped=0; query_time=0.000472; send_time=0.001472; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:38075>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.985106313Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5718 ( max 4 active 2 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.985123984Z condor_collector[5718]: (Sending 5 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.987997102Z condor_negotiator[51]: Sorting 5 ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.988083471Z condor_negotiator[51]: Got ads: 5 public and 3 private
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.988093779Z condor_negotiator[51]: Public ads include 0 submitter, 3 startd
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.988107945Z condor_negotiator[51]: Phase 2: Performing accounting ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.988513397Z condor_collector[5718]: Query info: matched=5; skipped=9; query_time=0.000533; send_time=0.003331; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:42877>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.998844673Z condor_negotiator[51]: Phase 3: Sorting submitter ads by priority ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.998907439Z condor_negotiator[51]: Starting prefetch round; 0 potential prefetches to do.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.998929547Z condor_negotiator[51]: Prefetch summary: 0 attempted, 0 successful.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.998946218Z condor_negotiator[51]: Phase 4.1: Negotiating with schedds ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.998962387Z condor_negotiator[51]: negotiateWithGroup resources used submitterAds length 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:58.998979147Z condor_negotiator[51]: ---------- Finished Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.991163407Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.991424377Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.992367538Z condor_collector[5719]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.992384393Z condor_collector[50]: QueryWorker: forked new worker with id 5719 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.992388706Z condor_collector[5719]: Query info: matched=3; skipped=0; query_time=0.000273; send_time=0.000240; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:46505>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.998996269Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:00.000539032Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.999281761Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.999295834Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:34:59.999690429Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000044; send_time=0.000042; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:46071>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:00.002235535Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.006619943Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.008400225Z condor_collector[5720]: (Sending 0 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.008427222Z condor_collector[50]: QueryWorker: forked new worker with id 5720 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.008432653Z condor_collector[5720]: Query info: matched=0; skipped=3; query_time=0.000315; send_time=0.000112; type=Machine; requirements={((Machine == "submit.pseven-htcondor"))}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.171:39583>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.008658384Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.009177019Z condor_collector[5721]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.009198843Z condor_collector[50]: QueryWorker: forked new worker with id 5721 ( max 4 active 1 pending 0 )
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.008399506Z condor_schedd[129]: Can't find address for startd submit.pseven-htcondor
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.008428458Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 91.5
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.008433979Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 91.5
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.008438073Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 91.5
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.008442150Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 91.5
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.008445835Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 91.5
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.010881594Z condor_schedd[129]: Found 3 potential dedicated resources in 0 seconds
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.011644531Z condor_schedd[129]: Adding submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor to the submitter map for default pool.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.010435082Z condor_collector[5721]: Query info: matched=3; skipped=0; query_time=0.000227; send_time=0.000795; type=Machine; requirements={((DedicatedScheduler == "DedicatedScheduler@parallel_schedd@submit.pseven-htcondor"))}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.171:43771>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.013354740Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.014554524Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.014576036Z condor_collector[5722]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.014581114Z condor_collector[50]: QueryWorker: forked new worker with id 5722 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.014890783Z condor_collector[5722]: Query info: matched=3; skipped=0; query_time=0.000355; send_time=0.000348; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:34097>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.022304168Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.023763143Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.022643762Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.022656155Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:01.023065283Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000035; send_time=0.000042; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:39969>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:01.024751347Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.051372463Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.052888527Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.055199120Z condor_collector[5723]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.055544310Z condor_collector[50]: QueryWorker: forked new worker with id 5723 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.058635829Z condor_collector[5723]: Query info: matched=3; skipped=0; query_time=0.001093; send_time=0.001196; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:45779>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.080369345Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:02.083500062Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:02.084834057Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.083485000Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.083524482Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:02.083536282Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000066; send_time=0.000101; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:44429>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.109659560Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.110405451Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.111615894Z condor_collector[5731]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.111644239Z condor_collector[50]: QueryWorker: forked new worker with id 5731 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.112314326Z condor_collector[5731]: Query info: matched=3; skipped=0; query_time=0.000603; send_time=0.000633; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:45591>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.122935714Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:03.124961392Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.123357886Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.123380410Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:03.123798350Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000043; send_time=0.000064; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:45443>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:03.125653791Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.157123216Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.158005912Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.162128169Z condor_collector[5732]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.162181769Z condor_collector[50]: QueryWorker: forked new worker with id 5732 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.162204861Z condor_collector[5732]: Query info: matched=3; skipped=0; query_time=0.001057; send_time=0.001065; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:39697>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.179947816Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:04.182532012Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.180401099Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.180535100Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:04.180559393Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000041; send_time=0.000070; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:33671>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:04.187025456Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.003785519Z condor_negotiator[51]: ---------- Started Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.003845884Z condor_negotiator[51]: Phase 1: Obtaining ads from collector ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.003860791Z condor_negotiator[51]: Getting startd private ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.005603360Z condor_collector[50]: Got QUERY_STARTD_PVT_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.007519651Z condor_collector[5733]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.007561310Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5733 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.008489720Z condor_collector[5733]: Query info: matched=3; skipped=0; query_time=0.001023; send_time=0.000827; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:34677>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.008856040Z condor_negotiator[51]: Getting Scheduler, Submitter and Machine ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.012807305Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5734 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.012862887Z condor_collector[5734]: (Sending 5 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.015750659Z condor_collector[5734]: Query info: matched=5; skipped=9; query_time=0.001216; send_time=0.002795; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:40391>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.018054423Z condor_negotiator[51]: Sorting 5 ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.018344335Z condor_negotiator[51]: Got ads: 5 public and 3 private
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.018381334Z condor_negotiator[51]: Public ads include 0 submitter, 3 startd
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.018399178Z condor_negotiator[51]: Phase 2: Performing accounting ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.023065518Z condor_negotiator[51]: Phase 3: Sorting submitter ads by priority ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.023162531Z condor_negotiator[51]: Starting prefetch round; 0 potential prefetches to do.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.023184553Z condor_negotiator[51]: Prefetch summary: 0 attempted, 0 successful.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.023200872Z condor_negotiator[51]: Phase 4.1: Negotiating with schedds ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.023216308Z condor_negotiator[51]: negotiateWithGroup resources used submitterAds length 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.023236995Z condor_negotiator[51]: ---------- Finished Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.209336230Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.210839752Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.212434071Z condor_collector[5735]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.212721734Z condor_collector[50]: QueryWorker: forked new worker with id 5735 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.213708655Z condor_collector[5735]: Query info: matched=3; skipped=0; query_time=0.001098; send_time=0.001061; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:40173>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.240038177Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:05.245817434Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.241687536Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.241743868Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:05.241796005Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000135; send_time=0.000195; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:40431>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:05.248110198Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.278875592Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.280027533Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.282073684Z condor_collector[5736]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.282321812Z condor_collector[50]: QueryWorker: forked new worker with id 5736 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.283180709Z condor_collector[5736]: Query info: matched=3; skipped=0; query_time=0.001074; send_time=0.000996; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:41745>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.303361713Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:06.306263770Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.303971736Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.305055746Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:06.305073498Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000070; send_time=0.000098; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:37603>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:06.308616499Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.333539809Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.335346066Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.336898437Z condor_collector[5737]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.337121225Z condor_collector[50]: QueryWorker: forked new worker with id 5737 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.337937232Z condor_collector[5737]: Query info: matched=3; skipped=0; query_time=0.000945; send_time=0.000930; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:33629>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.362025305Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:07.365626046Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.362900078Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.362955652Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:07.363149239Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000097; send_time=0.000124; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:40015>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:07.367230806Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.393829970Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.395081841Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.396875738Z condor_collector[5738]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.397205366Z condor_collector[50]: QueryWorker: forked new worker with id 5738 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.397890884Z condor_collector[5738]: Query info: matched=3; skipped=0; query_time=0.000928; send_time=0.000857; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:35749>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.421446806Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:08.427051755Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.422781289Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.422834728Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:08.423274922Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000106; send_time=0.000182; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:39023>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:08.429839795Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.459021979Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.460353492Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.461980538Z condor_collector[5739]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.462347289Z condor_collector[50]: QueryWorker: forked new worker with id 5739 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.462946748Z condor_collector[5739]: Query info: matched=3; skipped=0; query_time=0.000899; send_time=0.000858; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:42423>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.485793119Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:09.489813299Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.486690891Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.486725823Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:09.486906359Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000101; send_time=0.000125; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:41647>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:09.491640724Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.503797716Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.504026036Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.504929100Z condor_collector[5740]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.504994578Z condor_collector[50]: QueryWorker: forked new worker with id 5740 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.505173155Z condor_collector[5740]: Query info: matched=3; skipped=0; query_time=0.000315; send_time=0.000288; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:42455>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.514208409Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:10.515799207Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.514586278Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.514602384Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:10.514688130Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000049; send_time=0.000043; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:38963>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:10.516776384Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.030398787Z condor_negotiator[51]: ---------- Started Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.030423485Z condor_negotiator[51]: Phase 1: Obtaining ads from collector ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.030427518Z condor_negotiator[51]: Getting startd private ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.030805016Z condor_collector[50]: Got QUERY_STARTD_PVT_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.031593972Z condor_collector[5741]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.031613180Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5741 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.031748904Z condor_collector[5741]: Query info: matched=3; skipped=0; query_time=0.000263; send_time=0.000267; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:45665>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.031881625Z condor_negotiator[51]: Getting Scheduler, Submitter and Machine ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.032993518Z condor_collector[50]: QueryWorker: forked new high priority worker with id 5742 ( max 4 active 2 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.033006158Z condor_collector[5742]: (Sending 5 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.034027148Z condor_collector[5742]: Query info: matched=5; skipped=9; query_time=0.000297; send_time=0.000950; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.178:42863>; projection={}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.034617735Z condor_negotiator[51]: Sorting 5 ads ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.034718162Z condor_negotiator[51]: Got ads: 5 public and 3 private
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.034731673Z condor_negotiator[51]: Public ads include 0 submitter, 3 startd
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.034737942Z condor_negotiator[51]: Phase 2: Performing accounting ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.041833495Z condor_negotiator[51]: Phase 3: Sorting submitter ads by priority ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.041855224Z condor_negotiator[51]: Starting prefetch round; 0 potential prefetches to do.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.041873389Z condor_negotiator[51]: Prefetch summary: 0 attempted, 0 successful.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.041879584Z condor_negotiator[51]: Phase 4.1: Negotiating with schedds ...
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.041884199Z condor_negotiator[51]: negotiateWithGroup resources used submitterAds length 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.041890452Z condor_negotiator[51]: ---------- Finished Negotiation Cycle ----------
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.525998218Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.526160946Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.526850234Z condor_collector[5743]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.526863917Z condor_collector[50]: QueryWorker: forked new worker with id 5743 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.527138189Z condor_collector[5743]: Query info: matched=3; skipped=0; query_time=0.000340; send_time=0.000248; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:45507>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.535617595Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:11.537207027Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.535946729Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.535967578Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:11.536165132Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000035; send_time=0.000066; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:37997>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:11.538556827Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.563812514Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.564974655Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.566717030Z condor_collector[5744]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.567031951Z condor_collector[50]: QueryWorker: forked new worker with id 5744 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.567849213Z condor_collector[5744]: Query info: matched=3; skipped=0; query_time=0.000898; send_time=0.000886; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:40495>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.588149522Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:12.590814535Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.588750125Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.588773753Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:12.588830888Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000044; send_time=0.000081; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:45641>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:12.591953878Z condor_schedd[129]: Number of Active Workers 0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.619018102Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.620304880Z condor_collector[50]: Got QUERY_STARTD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.622075492Z condor_collector[5745]: (Sending 3 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.622228940Z condor_collector[50]: QueryWorker: forked new worker with id 5745 ( max 4 active 1 pending 0 )
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.622798588Z condor_collector[5745]: Query info: matched=3; skipped=0; query_time=0.000926; send_time=0.000633; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:33275>; projection={Activity Arch CondorLoadAvg EnteredCurrentActivity LastHeardFrom Machine Memory MyCurrentTime Name OpSys State}; filter_private_ads=0
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.637289447Z condor_collector[50]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:13.639588609Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap.
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.637710586Z condor_collector[50]: Got QUERY_SCHEDD_ADS
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.637734180Z condor_collector[50]: (Sending 1 ads in response to query)
[pod/pseven-htcondormanager-deploy-85686d5d6-s5g9r/htcondormanager] 2021-06-15T15:35:13.638217594Z condor_collector[50]: Query info: matched=1; skipped=0; query_time=0.000048; send_time=0.000064; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.171:45583>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0
[pod/pseven-htcondorsubmit-deploy-55c56869fd-jzmkm/htcondorsubmit] 2021-06-15T15:35:13.640883493Z condor_schedd[129]: Number of Active Workers 0
^C
â??-INT
_________________________________________________________________________________________________________________________________[15:35:14]
[coder@devenv: /pSeven/] [py:venv] [k8s:default] [1038-htcondor-improvements-v2 â??·47â??·36|â?? 5â?¦3â?? 60]
>>>
-- Schedd: parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx : <10.42.0.171:38121?...
The Requirements expression for job 91.000 is
((NumJobStarts == 0) && (OpSys == "LINUX") && (Arch == "X86_64") && (DA__P7__CLUSTER_NODE == "True")) &&
(TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && (TARGET.Cpus >= RequestCpus) &&
((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
Job 91.000 defines the following attributes:
DiskUsage = 0
FileSystemDomain = "pseven-htcondor"
NumJobStarts = 1
RequestCpus = 25
RequestDisk = DiskUsage
RequestMemory = 256
The Requirements expression for job 91.000 reduces to these conditions:
Slots
Step Matched Condition
----- -------- ---------
[0] never NumJobStarts == 0
Last successful match: Tue Jun 15 14:55:40 2021
091.000: Run analysis summary ignoring user priority. Of 3 machines,
3 are rejected by your job's requirements
0 reject your job because of their own requirements
0 match and are already running your jobs
0 match but are serving other users
0 are able to run your job
WARNING: Be advised:
No machines matched the jobs's constraints
The Requirements expression for job 91.001 is
((NumJobStarts == 0) && (OpSys == "WINDOWS" || OpSys == "LINUX") && (Arch == "X86_64") && (DA__P7__RUNENV_PYTHON3 >= 13) &&
(DA__P7__CLUSTER_NODE == "True")) && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) &&
((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
Job 91.001 defines the following attributes:
DiskUsage = 42
FileSystemDomain = "pseven-htcondor"
NumJobStarts = 0
RequestDisk = DiskUsage
RequestMemory = 128
The Requirements expression for job 91.001 reduces to these conditions:
Slots
Step Matched Condition
----- -------- ---------
[2] 3 OpSys == "LINUX"
[5] 3 Arch == "X86_64"
[7] 3 DA__P7__RUNENV_PYTHON3 >= 13
[9] 3 DA__P7__CLUSTER_NODE == "True"
[11] 3 TARGET.Disk >= RequestDisk
[13] 3 TARGET.Memory >= RequestMemory
[15] 3 TARGET.FileSystemDomain == MY.FileSystemDomain
No successful match recorded.
Last failed match: Tue Jun 15 14:55:24 2021
Reason for last match failure: PREEMPTION_REQUIREMENTS == False
091.001: Run analysis summary ignoring user priority. Of 3 machines,
0 are rejected by your job's requirements
0 reject your job because of their own requirements
0 match and are already running your jobs
0 match but are serving other users
3 are able to run your job
The Requirements expression for job 91.002 is
((NumJobStarts == 0) && (OpSys == "WINDOWS" || OpSys == "LINUX") && (Arch == "X86_64") && (DA__P7__RUNENV_PYTHON3 >= 13) &&
(DA__P7__CLUSTER_NODE == "True")) && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) &&
((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
Job 91.002 defines the following attributes:
DiskUsage = 42
FileSystemDomain = "pseven-htcondor"
NumJobStarts = 0
RequestDisk = DiskUsage
RequestMemory = 128
The Requirements expression for job 91.002 reduces to these conditions:
Slots
Step Matched Condition
----- -------- ---------
[2] 3 OpSys == "LINUX"
[5] 3 Arch == "X86_64"
[7] 3 DA__P7__RUNENV_PYTHON3 >= 13
[9] 3 DA__P7__CLUSTER_NODE == "True"
[11] 3 TARGET.Disk >= RequestDisk
[13] 3 TARGET.Memory >= RequestMemory
[15] 3 TARGET.FileSystemDomain == MY.FileSystemDomain
No successful match recorded.
Last failed match: Tue Jun 15 14:55:24 2021
Reason for last match failure: PREEMPTION_REQUIREMENTS == False
091.002: Run analysis summary ignoring user priority. Of 3 machines,
0 are rejected by your job's requirements
0 reject your job because of their own requirements
0 match and are already running your jobs
0 match but are serving other users
3 are able to run your job
The Requirements expression for job 91.003 is
((NumJobStarts == 0) && (OpSys == "WINDOWS" || OpSys == "LINUX") && (Arch == "X86_64") && (DA__P7__RUNENV_PYTHON3 >= 13) &&
(DA__P7__CLUSTER_NODE == "True")) && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) &&
((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
Job 91.003 defines the following attributes:
DiskUsage = 42
FileSystemDomain = "pseven-htcondor"
NumJobStarts = 0
RequestDisk = DiskUsage
RequestMemory = 128
The Requirements expression for job 91.003 reduces to these conditions:
Slots
Step Matched Condition
----- -------- ---------
[2] 3 OpSys == "LINUX"
[5] 3 Arch == "X86_64"
[7] 3 DA__P7__RUNENV_PYTHON3 >= 13
[9] 3 DA__P7__CLUSTER_NODE == "True"
[11] 3 TARGET.Disk >= RequestDisk
[13] 3 TARGET.Memory >= RequestMemory
[15] 3 TARGET.FileSystemDomain == MY.FileSystemDomain
No successful match recorded.
Last failed match: Tue Jun 15 14:55:24 2021
Reason for last match failure: PREEMPTION_REQUIREMENTS == False
091.003: Run analysis summary ignoring user priority. Of 3 machines,
0 are rejected by your job's requirements
0 reject your job because of their own requirements
0 match and are already running your jobs
0 match but are serving other users
3 are able to run your job
The Requirements expression for job 91.004 is
((NumJobStarts == 0) && (OpSys == "WINDOWS" || OpSys == "LINUX") && (Arch == "X86_64") && (DA__P7__RUNENV_PYTHON3 >= 13) &&
(DA__P7__CLUSTER_NODE == "True")) && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) &&
((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
Job 91.004 defines the following attributes:
DiskUsage = 42
FileSystemDomain = "pseven-htcondor"
NumJobStarts = 0
RequestDisk = DiskUsage
RequestMemory = 128
The Requirements expression for job 91.004 reduces to these conditions:
Slots
Step Matched Condition
----- -------- ---------
[2] 3 OpSys == "LINUX"
[5] 3 Arch == "X86_64"
[7] 3 DA__P7__RUNENV_PYTHON3 >= 13
[9] 3 DA__P7__CLUSTER_NODE == "True"
[11] 3 TARGET.Disk >= RequestDisk
[13] 3 TARGET.Memory >= RequestMemory
[15] 3 TARGET.FileSystemDomain == MY.FileSystemDomain
No successful match recorded.
Last failed match: Tue Jun 15 14:55:24 2021
Reason for last match failure: PREEMPTION_REQUIREMENTS == False
091.004: Run analysis summary ignoring user priority. Of 3 machines,
0 are rejected by your job's requirements
0 reject your job because of their own requirements
0 match and are already running your jobs
0 match but are serving other users
3 are able to run your job
Every 1.0s: condor_status; condor_q -all -global pseven-htcondorsubmit-deploy-55c56869fd-jzmkm: Tue Jun 15 15:34:09 2021
Name OpSys Arch State Activity LoadAv Mem ActvtyTime
slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LINUX X86_64 Unclaimed Idle 0.000 1800 0+01:04:08
slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LINUX X86_64 Unclaimed Idle 0.000 1800 0+01:04:09
slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx LINUX X86_64 Unclaimed Idle 0.000 1800 0+01:04:09
Total Owner Claimed Unclaimed Matched Preempting Backfill Drain
X86_64/LINUX 3 0 0 3 0 0 0 0
Total 3 0 0 3 0 0 0 0
-- Schedd: parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx : <10.42.0.171:46693?... @ 06/15/21 15:34:09
OWNER BATCH_NAME SUBMITTED DONE RUN IDLE HOLD TOTAL JOB_IDS
user20001 ID: 91 6/15 14:53 _ _ 5 _ 5 91.0-4
Total for query: 5 jobs; 0 completed, 0 removed, 5 idle, 0 running, 0 held, 0 suspended
Total for all users: 5 jobs; 0 completed, 0 removed, 5 idle, 0 running, 0 held, 0 suspended