Re: [HTCondor-users] Condor bug matching with an offline machine

[Jens Schmaler <jens.schmaler@xxxxxx>]

Dear Todd and John,

let me add a little more background here:

Our non-dedicated machines are sometimes switched off by their owners -
we do not have HTCondor configured to do that. HTCondor then treats them
as "absent" at first, but we have a cron job configured which regularly
advertises the absent class ads as "offline", because we want HTCondor
to wake them up upon need.

As described by Almansour, the problem arises when wake-on-LAN is not
successful for one of those machines. In spite of the fact that we have
the "black hole policy" in place as you suggest, for some reason
HTCondor sometimes keeps on matching the job to the same (offline)
machine over and over again, although other machines would be available.
So I am wondering

- whether there is a possibility that the LastMatchName somehow is not
updated, so that the same machine may be matched again.

- why the job is at all matched to an offline machine, in spite of the
fact that online machines would be available.

Do you think there might be a bug here?

@John: We have often had trouble with black hole machines, so any
improvement in handling this would be highly appreciated. We would
definitely be interested to discuss about possible changes here!

Thanks and best regards,
Jens



Am 09.03.17 um 09:37 schrieb Almansour Blanco:
> Hello,
> 
> The machines we have that are offline are machines that have wake on
> lan problem We keep them in the list in order to be woken up
> eventually, so that when we change something in the BIOS, there might
> be a possibility that the machine wakes up. As we add more and more
> machines, the problem is always prone to happen again, and have an
> offline machine that will become a black hole.
> 
> Beste regards,
> 
> Almansour Belleh Blanco
> 
> 
> 
> -----Original Message----- From: HTCondor-users
> [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of John M
> Knoeller Sent: Freitag, 3. März 2017 19:23 To: HTCondor-Users Mail
> List <htcondor-users@xxxxxxxxxxx> Subject: Re: [HTCondor-users]
> Condor bug matching with an offline machine
> 
> I'm not sure I understand, why would you put machines in the offline
> state (instead of just removing them from the collector entirely) if
> you don't want them to match and get revived when they are needed to
> run jobs?
> 
> Does the fact they are in the collector but offline serve some other
> purpose for your pool?
> 
> Also, I wonder if you would have some time to discuss a possible
> improved interface for dealing with blackhole machines. We are
> considering making tagging a machine as a blackhole a first -class
> operation in HTCondor, so that you would be Able to do it without
> changing any job's requirements expression or removing the machines
> from the collector entirely.
> 
> For instance, maybe you could send a command to the schedd to tell it
> to not accept matches with a machine.  Would That be an acceptable
> solution or part of a solution to your current problem?
> 
> -tj
> 
> -----Original Message----- From: HTCondor-users
> [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Almansour
> Blanco Sent: Thursday, March 2, 2017 9:18 AM To: HTCondor-Users Mail
> List <htcondor-users@xxxxxxxxxxx> Subject: Re: [HTCondor-users]
> Condor bug matching with an offline machine
> 
> Hello,
> 
> Thank you for your response. We already have the blackhole
> configuration in place, so the jobs always try in another machine. 
> What I think is a bug is that it should not match with an offline
> machine in first place
> 
> Kind regards,
> 
> 
> -----Original Message----- From: HTCondor-users
> [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Todd
> Tannenbaum Sent: Donnerstag, 2. März 2017 14:26 To: HTCondor-Users
> Mail List <htcondor-users@xxxxxxxxxxx> Subject: Re: [HTCondor-users]
> Condor bug matching with an offline machine
> 
> On 3/2/2017 4:44 AM, Almansour Blanco wrote:
>> Hello,
>> 
>> I am using condor 8.4.3 on windows 7 64 bits.
>> 
>> I have a strange bug.
>> 
>> We have some machines that have Wake on LAN  problems in the
>> network.
>> 
>> What happens is, when a job is matched to one of these machines
>> while it is offline, it tries to wake it up, which fails of
>> course.
>> 
>> On the next negotiation cycle, the same job can be matched to this
>>  machine again, and this keeps on happening again and again.
>> 
>> The normal course of action, as far as I know, when a job is
>> matched to a machine and it fails, it will always try with another
>> machine, which doesn't seem to be the case here.
> 
> What is putting the machine to sleep such that it needs to be waken
> up over the LAN?  Is it some screen saver, or HTCondor itself via the
> HIBERNATE expression in the condor_config file?  I am guessing it is
> a screen saver or some such, as if it was HTCondor itself the machine
> classads would be tagged as offline and would not be matched until
> successfully woken.  So perhaps one idea is to use HTCondor for your
> power management and let it control when to put machines offline;
> see
> 
> http://htcondor.org/manual/current/3_18Power_Management.html
> 
> When a job is matched to a machine and fails, it will not necessarily
> try another machine - it may, as you observe, try the same machine
> again.  This is something we should consider improving in a future
> release, however, for now you can use job policy expressions in your
> job's condor_submit file to achieve this.  For an example of how to
> do this, see the HOWTO recipes, specifically
> 
> https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=AvoidingBlackHoles
>
>  Hope the above helps Todd
> 
> _______________________________________________ HTCondor-users
> mailing list To unsubscribe, send a message to
> htcondor-users-request@xxxxxxxxxxx with a subject: Unsubscribe You
> can also unsubscribe by visiting 
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________ HTCondor-users
> mailing list To unsubscribe, send a message to
> htcondor-users-request@xxxxxxxxxxx with a subject: Unsubscribe You
> can also unsubscribe by visiting 
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________ HTCondor-users
> mailing list To unsubscribe, send a message to
> htcondor-users-request@xxxxxxxxxxx with a subject: Unsubscribe You
> can also unsubscribe by visiting 
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________ HTCondor-users
> mailing list To unsubscribe, send a message to
> htcondor-users-request@xxxxxxxxxxx with a subject: Unsubscribe You
> can also unsubscribe by visiting 
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/htcondor-users/
>