Re: [HTCondor-users] How to use condor_chirp?

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Brian,

thanks for the warning.

Our idea is to update a job's ClassAd with information on the cgroups
context its slot is using.
In the end, we would like to to be able to refer a job to the cgroups
statistics, i.e., to monitor the local cgroups and send statistics into
an InfluxDB and in parallel accumulate basic job information from Condor
(frequency, scaling, only accumulation? etc. remains to be seen...).

In the end, it would be nice to be able to plot basic jobs statistics
from InfluxDB with Grafana for at least a subset of jobs (individual
users - large scale/institutional users may be not necessary) for a
range of a few days (regular pruning of data points/measurements in
InfluxDB - scales how well?).

One question for me is, if such condor job statistics are better
accumulated/send from the schedd or better (somehow?) from the startds?
For cgroup statistics I would try to send them directly from the nodes
(cAdvisor or similar approach).

Cheers,
  Thomas

On 2016-03-15 12:23, Brian Bockelman wrote:
> Hi Thomas,
> 
> Two things to note:
> 1) "condor_chirp set_job_attrâ requires +WantIOProxy=true in the classad.  This updates the ClassAd immediately in the schedd - which can be a scalability concern.
> 2) âcondor_chirp set_job_attr_delayedâ works by default.  However, it only sends the attribute in the next scheduled ClassAd update (for CPU and memory usage); thereâs less scalability concern.  Additionally, attributes set with this command must start with the prefix âChirpâ (case-sensitive).
> 
> Can you give some background on what youâre trying to accomplish?
> 
> Brian
> 
>> On Mar 14, 2016, at 12:44 PM, Thomas Hartmann <thomas.hartmann@xxxxxxx> wrote:
>>
>> Hi all,
>>
>> I would like to inject some system information into jobs' ClassAds.
>> As I understand condor_chirp, I cannot inject/manipulate ClassAds on a
>> worker for a running job, but only during submission. (I assume, that
>> ~/.job.ad is where a job keeps its ClassAds [1] -- but can I inject it
>> from outside the job 'properly'?)
>>
>> We get our grid jobs currently via an ARC-CE so I suppose the best place
>> would be there (where?) to enable the communication token by appending
>> +WantIOProxy = TRUE
>> to job submissions scripts, or?
>>
>> I am not sure, where to actually call condor_chirp on the worker?
>> My best guess is so far, to find the template for the job wrappers, i.e.,
>> /var/lib/condor/execute/dir_*/condor_exec.exe
>> and chirp from it, or? Probably starting a separate thread for updating
>> changing ClassAds.
>> Would that be reasonable or is there a better way?
>> Where would I find the wrapper for condor_exec.exe?
>>
>> Cheers and thanks,
>>  Thomas
>>
>>
>>
>> [1]
>>> cat /proc/`ps axf | grep "/bin/bash -l /var/lib/condor/execute/dir" |
>> tail -n1 | cut -d " " -f 2`/environ  | tr '\0' '\n' | grep "_CONDOR_JOB_AD"
>> _CONDOR_JOB_AD=/var/lib/condor/execute/dir_38259/.job.ad
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature