[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Preferred htcss installation method



Hi Todd,

Yeah, this was exactly what I was trying to understand and you walked me through it with a clarity that I don't have much else to say. Thanks!

Cheers,
Matt

Matthew T. West
DevOps & HPC SysAdmin
University of Exeter, Research IT
57 Laver Building, North Park Road, Exeter, EX4 4QE, United Kingdom

On 26/06/2024 23:11, Todd L Miller wrote:
CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe.


But I have to wonder, what improvement to the overall throughput of the
pool would you get by using a tuned set of compile flags for that
microarchitecture rather than a provided binary? Any at all?

ÂÂÂÂÂÂ I would expect the gains, if any, to be extremely marginal, except
in unusual situations. Assuming that "the throughput of the pool" means,
let's say, how many jobs complete per week, we can try to guess at how
many more jobs would complete by determining how much shorter the interval
between job starts on the same resources would be if HTCondor didn't use
any CPU time at all. (I ignore resources used by the AP and CM because
they don't run jobs in the pools I'm familiar with.)

 So: a job in a slot completes. If the slot is re-used, it doesn't
take HTCondor very long at all to start transferring the next job -- I
would expect on the order of seconds at the most, with most of that time
being trying ot get the schedd's attention. Assuming an average job
length of an hour, even if it takes ten seconds to start transferring the
next job, that's less than .3% extra time that could be gained.

ÂÂÂÂÂÂ The more interesting case would be when the slot can't be re-used
and has to be renegotiated. The negotiator certainly uses a fair amount
of CPU, but my impression was that most of the length of any given
negotiator cycle was taken up by i/o of various kinds. Ignoring that,
even on fairly large pools, a negotiation cycle usually takes less than a
minute, so even if each slot finished just after a negotiation cycle
started and thus didn't become active until just after the end of the next
one, you're looking at ~2/62, or saving ~3% even if the negotiation cycle
took no time at all.

ÂÂÂÂÂÂ I would expect the largest possible gain from either case to be
much smaller in practice, and of course, only a fraction of that gain can
be unlocked by compiler optimizations.

-- ToddM