[Condor-users] VMs being cleaned up/removed

[Leslie Groer <groer@xxxxxxxxxxxxxxxxxxx>]

I am running Condor version 6.7.2 on Scientific Linux 3.0.3
with 11 dual-cpu worker nodes with 4 VMs each.  There are three schedulers
and the CM is using kerberos authentication.

I notice that fairly often, VMs will be "cleaned up" during housecleaning 

============== CollectorLog =============================
4/26 09:56:19 Housekeeper:  Ready to clean old ads
4/26 09:56:19 	Cleaning StartdAds ...
4/26 09:56:19 		**** Removing stale ad: "< 
vm4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.0.11.16 >"
4/26 09:56:19 	Cleaning StartdPrivateAds ...
4/26 09:56:19 		**** Removing stale ad: "< 
vm3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.0.11.25 >"
4/26 09:56:19 	Cleaning ScheddAds ...
4/26 09:56:19 	Cleaning SubmittorAds ...
4/26 09:56:19 	Cleaning LicenseAds ...
4/26 09:56:19 	Cleaning MasterAds ...
4/26 09:56:19 	Cleaning CkptServerAds ...
4/26 09:56:19 	Cleaning CollectorAds ...
4/26 09:56:19 	Cleaning StorageAds ...
4/26 09:56:19 Housekeeper:  Done cleaning
=========================================================

Some of these VMs reappear later on, but there are times were almost all
the VMs are removed, even though there is no load on the system.  This
causes problems for Grid applications as our site needs to publish
available resources which currently is based on total number of VMs and
the number being used.  I have yet to find a cause for the stale ads.  Is
this a Condor "feature" or is this indicative of some problem?  I have
used 'condor_restart -all' to get all the VMs back, which seems a bit
harsh, especially if there are jobs in the system.  'condor_reconfig' does
not seem to do anything.

Thanks
Leslie Groer

-- 
   ,-~~-.___.       ________________________________________________
  / |  '     \      groer@xxxxxxxxxxxxxxxxxxx  Department of Physics
 (  )        0           Tel: +1-416-978-2959  University of Toronto
  \_/-, ,----'           Fax: +1-416-978-8221  60 St. George Street
     ====           //                         Toronto, ON M5S 1A7
    /  \-'~;    /~~~(O)                        Canada
   /  __/~|   /       |  Office: McLennan Physics Lab Room 911
 =(  _____| (_________|  http://home.fnal.gov/~groer
     Leslie S. Groer