[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Delay in job log writing

On 8/15/2013 9:15 AM, Smith, Ian wrote:
No answer to thisunfortunately but just to confirm that I have also seem this (with
version 7.8.6)
> and it is a bit frustrating if you are polling the log files for events rather than
> using condor_q (not a good idea as it can overload the scheduler).
> My guess is that Condor only updates the logfiles periodically and this isn't down > to an fsync() problem. If that's the case, is there any way to configure Condor to
> update the logfiles more often ?

There is an answer - I sent the answer to the list on Monday and the original poster confirmed things are working as I explained.

Post from Monday is here:

Relevant info from the post is:

The condor_q tool reports a job as "running" as soon as a condor_shadow process is spawned for the job. However, the execute event does not show up in the event log until the moment before the job is spawned on the remote execute machine. Thus the time it takes the shadow to activate the claim on the execute machine (aka start a condor_starter) and potentially transfer any input files will show up as running time with condor_q, but an execute event will not appear in the log until these operations complete.

Hope the above helps,



From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of ???
Sent: 12 August 2013 15:17
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Delay in job log writing

Hi, I find that there is delay in job log file writing. In my test, condor_q showed one job were running, but there were only a submit event of it in the log file. After a while, more events occurs and they would be wrote together. I think this is because of the buffer in write/fwrite function.
I hope that my log reader program can know events as soon as possible, so is there  any configuration can do this? I think condor_shadow should fsync after writing one event into log file.

HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685