Hi,
I'm documenting here my recipe to solve my installation issues,Âif there is a better forum to contribute to HTCondor, please let me know. Also, please feel free to let me know if something here is wrong or could be improved. I hope this helps someone else google'ingÂin the future.Â
Task : Upgrade Condor 7.xx to 9.1 and do breadth first load balancing
ProblemÂ1 -ÂHtCondor installation script won't complete because previous condor install not completelyÂremoved with apt purge
What: HtCondor installation script keeps failing
Why: Previous install not completely removed via apt purge
Remedy: ManuallyÂremove everything
# Remove old condor user no longer needed
sudo deluser --remove-home condor
# Purge Condor
sudo apt purge condorÂÂ
# Ensure systemctl services and entries are stopped and removed respectively
systemctl stop condor
systemctl disable condor
rm /etc/systemd/system/condor
rm /etc/systemd/system/condor.service
rm /usr/lib/systemd/system/condor
systemctl daemon-reload
systemctl reset-failed
Problem 2 - Condor detectingÂall execute nodes as localhost because of /etc/hosts format
What: Condor sees all execute hosts as slot1@localhost, slot2@localhost, etcÂ
Why: /etc/hosts needs to conform to a specific format so that Condor's heuristic method for determining hostname and domainameÂoperate correctly
Remedy: alter /etc/hostsÂ
Specifically, the loopback address cannot contain the machinenameÂelse condor associates "localhost" as the "machinename".ÂÂ
ie this is bad
127.0.0.1 localhost machinename
127.0.1.1 machinename.domain.name machinename
192.168.7.1 machinename
and this is good...
127.0.0.1 localhost
127.0.1.1 machinename.domain.name machinename
192.168.7.1 machinename
I feel this may well be something limiting in condor. Given we use docker and containers and can have multiple machines names on the same ip, it's not clear how condor will resolve this in a more complicated setup.
See alsoÂ
Problem 3 - Jobs always running as "nobody" and not running as submitting "user" because UID_DOMAIN always differs on submit and execute machines in default setup due to our (no domain) networkÂsetupÂ
What : Files written to shared file system are written as nobody:nouser (user:group) which makes them unreadable to the submitter.
Why: Default UID_DOMAIN in condor install is $(FULL_HOSTNAME) which differs for every machine by default in our setup. Further, the submitter domain looks different on it;s own machine because we had no domainname in our ubuntu setup
Remedy: Override UID_DOMAIN on every machine by adding custom config in /etc/condor/config.d/99_custom.config on every machine
# Make sure every machine knows it's on the same
# user id domain and thus can run the job as the submitting user (eg lyle)
# we do not use domainnamesÂon our internal network so set a default one
DEFAULT_DOMAIN_NAME=ocwen.com
# For the UID_DOMAIN to be the same on all machines UID_DOMAIN=ocwen.com
# Don't check submitter's UID_DOMAIN, just trust it TRUST_UID_DOMAIN=true
SOFT_UID_DOMAIN=true
Note that I went pretty hard here to force condor to match UID_DOMAIN.
Problem 4 - Condor by default allocates resources to the best machine first, then second best machine and so on (aka "depth first"). We would rather condor spread jobs over the best machines (aka "breadth first").
What: If two machines have slightlyÂdifferent KFlops and many slots, potentially all slots for one machine will first before the other
Why: DefaultÂNEGOTIATOR_PRE_JOB_RANK setting
Remedy: ModifyÂNEGOTIATOR_PRE_JOB_RANK for breadth first allocation to slots on a machine inÂ/etc/condor/config.d/99_custom.configÂ
NEGOTIATOR_PRE_JOB_RANK = ( $(NEGOTIATOR_PRE_JOB_RANK) ) * (SlotID/ÂTotalSlots)
Problem 5 - Default condor install allows only 1 role for a machine
What : Submitter machine wont be an excute machine by default. Obviously sharing is caring here.
Why: Default DAEMON_LIST from the meta-config is geared towards one role
Remedy : Add the required dameon in /etc/condor/config.d/99_custom.configÂ
# Make Every Condor submit machine also an execute node
DAEMON_LIST=$(DAEMON_LIST) STARTD
Finally... Problem 6 - How to add nodes fast
We use jumpcloud for SSO on all machines which has a nifty tool to run commands remotely as root according to defined "groups" of devices. So to add a amchine to our pool all we do is
curl -fsSL https://get.htcondor.org | GET_HTCONDOR_PASSWORD="$htcondor_password" sudo /bin/bash -s -- --no-dry-run --submit $central_manager_name
cat <<EOL > /etc/condor/config.d/02_aec_custom.config && condor_reconfig && mount -a
# Make sure every machine knows it's on the same
# user id domain and thus can run the job as the submitting user (eg lyle)
DEFAULT_DOMAIN_NAME=ocwen.com
UID_DOMAIN=ocwen.com
TRUST_UID_DOMAIN=true
SOFT_UID_DOMAIN=true
# Make Every Condor machine also an execute node
DAEMON_LIST=\$(DAEMON_LIST) STARTD
# Define Load Balancing on the AE Pool
NEGOTIATOR_PRE_JOB_RANK=( \$(NEGOTIATOR_PRE_JOB_RANK) ) * (SlotID/ TotalSlots)
EOL
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}