SYSTEM_JOB_MACHINE_ATTRS is a list of Machine attributes copied from the match ad (which the Negotiator sends to the Schedd) into the Job ad when a job starts running. This is something that the Schedd does and only when a job starts.
A change in the execute node of the value of the attribute that SYSTEM_JOB_MACHINE_ATTRS is copying will not be reflected into jobs until the next time a job starts on that machine *as the result of a full negotiation cycle*, so this can take a very long time to propagate, and the value will never change while a job a running.
For something like node health, which can change as the job runs, I think you want to configure STARTD_JOB_ATTRS on the execute node instead of SYSTEM_JOB_MACHINE_ATTRS on the submit node.
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx>
On Behalf Of ervikrant06@xxxxxxxxx
We are running condor jobs on pre-emptible google cloud instances. I wanted to add something in job classad to identify the jobs impacted because of pre-empted instances.
On sched file:
SYSTEM_JOB_MACHINE_ATTRS = $(SYSTEM_JOB_MACHINE_ATTRS) nodehealth
on started classAD is advertised.
test.example:/etc/condor/config.d# condor_status -compact `hostname` -af machine nodehealth
I can see the following in job classAD.
$ condor_q -run -af jobruncount MachineAttrnodehealth0 MachineAttrnodehealth1
But when I change the value of classAD (by directly modifying condor configuration and running condor_reconfig) on executor node it's not getting reflected in job definition.
I have seen this message in log file. Our executor directory is only
06/02/20 06:49:38 slot1_1: Failed to open '/spare/condor/dir_418909/.update.ad.tmp' for writing update ad: No such file or directory (2).
However I do see that .updated.ad file inside the execution directory has the updated value but still machine and job ad reflecting old value as they can't change dynamically.
# grep nodehealth .update.ad
# grep nodehealth .job.ad
# grep nodehealth .machine.ad
# condor_status -compact `hostname` -af machine nodehealth
After hold/release job is picking new value but I want to update the value in running instance of job.
gone through link  but that one also is not useful.
Any input is highly appreciated.
Thanks & Regards,