[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Adding custom job classads on condor_starter nodes

SYSTEM_JOB_MACHINE_ATTRS is a list of Machine attributes copied from the match ad (which the Negotiator sends to the Schedd) into the Job ad when a job starts running.  This is something that the Schedd does and only when a job starts. 


A change in the execute node of the value of the attribute that SYSTEM_JOB_MACHINE_ATTRS is copying will not be reflected into jobs until the next time a job starts on that machine *as the result of a full negotiation cycle*, so this can take a very long time to propagate, and the value will never change while a job a running.


For something like node health, which can change as the job runs, I think you want to configure STARTD_JOB_ATTRS on the execute node instead of SYSTEM_JOB_MACHINE_ATTRS on the submit node.




From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of ervikrant06@xxxxxxxxx
Sent: Tuesday, June 2, 2020 6:40 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Adding custom job classads on condor_starter nodes


Hello Experts, 


We are running condor jobs on pre-emptible google cloud instances. I wanted to add something in job classad to identify the jobs impacted because of pre-empted instances. 


On sched file: 




on started classAD is advertised. 


test.example:/etc/condor/config.d# condor_status -compact `hostname` -af machine nodehealth
test.example.com False1


I can see the following in job classAD. 


$ condor_q -run -af jobruncount MachineAttrnodehealth0 MachineAttrnodehealth1
1 False1 undefined
1 False1 undefined


But when I change the value of classAD (by directly modifying condor configuration and running condor_reconfig) on executor node it's not getting reflected in job definition. 


I have seen this message in log file. Our executor directory is only 


06/02/20 06:49:38 slot1_1: Failed to open '/spare/condor/dir_418909/.update.ad.tmp' for writing update ad: No such file or directory (2).


However I do see that .updated.ad file inside the execution directory has the updated value but still machine and job ad reflecting old value as they can't change dynamically. 


# grep nodehealth .update.ad
nodehealth = "False4"


# grep nodehealth .job.ad
MachineAttrnodehealth0 = "False1"


# grep nodehealth .machine.ad
nodehealth = "False1"


# condor_status -compact `hostname` -af machine nodehealth
test.example.com False4


After hold/release job is picking new value but I want to update the value in running instance of job. 


gone through link [1] but that one also is not useful.


Any input is highly appreciated. 


Thanks & Regards,

Vikrant Aggarwal