Re: [Condor-users] about multiple scheduler issues

["Carson Hung" <carson@xxxxxxxxxxxxxx>]

Sorry for sending 2 emails on replies, as I suddenly thought of somethings
and would like to know.

In the case of matchmaking in condor-g, is preemption works and possible
to occur?

Thanks,
Carson

> Thanks very much for your clear explanation.
> I will setup a simple experiment in lab to test it.
>
> One more additional question (really sorry for many questions).
>
> In the negotiation cycle, the negotiator will match the resources to
> users in what order. what I mean here is that, if I have 3 grid sites,
> each can accommodate 3 jobs, and I have 3 users with infinity jobs
> queuing.
>
> Suppose they are all with the same user priority and job priority. In
> this case, so theoretically, each users should be able to submit 3 jobs
> to these 3 grid sites. But the job of user 1 will all be matched to grid
> site 1 or distributed to 1 to each grid site.
>
> Thanks,
> Carson
>
>>
>>
>> Carson Hung wrote:
>>
>>>I think what Erik's mean in replacing claiming of resources in
>>> condor-g
>>> is by setting the CurMatches and WantAdRevaluate, right?
>>>
>>
>> Yes.
>>
>>>However, I am not very understand why Dagman is necessary in this
>>> case?
>>>
>>
>> DAGMan is not necessary in Condor-G matchmaking. DAGMan is used by
>> USCMS to coordinate the different stages of a job (stage-in, run,
>> stage-out, cleanup).
>>
>>>and in Dagman, all the jobs that is needed to be executed will be
>>> pre-grouped into certain size and managed into a Dag file. In this
>>> case, all the member of jobs in the same group can only be submited
>>> to the same site, right?
>>>
>>
>> Yes. Multiple DAGs are submitted. The matchmaking process directs them
>> to target different sites, so each of the jobs within the DAGs are
>> directed to the same site. This is just one way of doing things. It's
>> not the only way.
>>
>>> and the decision will be done in the negotiator site, but not
>>>the matchmaker site, right? is there any method that I can change the
>>> scheduling based on the conditions of grid sites in matchmaker site?
>>>
>>
>> The negotiator and the matchmaker are the same thing. The system
>> described in "USCMS Matchmaking" is sensitive only to the number of
>> jobs we have submitted to a given site and the maximum number of jobs
>> we have configured for that site. To first order, this is a good way
>> of doing matchmaking between grid sites. To second order, you may also
>> want to monitor live queue depths at remote sites and feed this back
>> into the site ClassAds, perhaps to lower the rank of busy sites and
>> increase the rank of non-busy ones. However, I don't consider it
>> necessary to worry about the second order optimizations until you have
>> the first order ones working:)
>>
>> --Dan
>>
>>>>http://www.cs.wisc.edu/condor/USCMS_matchmaking.html
>>>>
>>>>Carson Hung wrote:
>>>>
>>>>
>>>>
>>>>>Hi Erik,
>>>>>
>>>>>One more question to ask, where can I find the USCMS writeup :)
>>>>>
>>>>>Thanks,
>>>>>Carson
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Hi Erik,
>>>>>>
>>>>>>Thanks very much for your clear explanation.
>>>>>>
>>>>>>You are right, I am actually asking if it is possible to have more
>>>>>> than one negotiators with multiple schedulers. Since I think that
>>>>>> case is possible to exist in condor-g environment.
>>>>>>
>>>>>>and I think you have answered my questions. Really thanks a lot.
>>>>>>
>>>>>>Thanks,
>>>>>>Carson
>>>>>>
>>>>>>
>>>>>>I have more understanding about the condor and condor-g
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>On Sat, Feb 19, 2005 at 04:40:24PM +0800, Carson Hung wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>Hi,
>>>>>>>>
>>>>>>>>Since it is possible to have more than one schedulers/submittors
>>>>>>>> in the clusters.
>>>>>>>>So I would like to know if it is possible that 2 or more
>>>>>>>> schedulers match the same resource to the different jobs if they
>>>>>>>> come from different schedulers/submittors.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>I think you really mean "if it is possible that 2 or more
>>>>>>> negotiators match the same resources to different jobs if they
>>>>>>> come from
>>>>>>>different schedds" - you can't use the term 'scheduler' twice :)
>>>>>>>
>>>>>>>In Condor, it's not really possible for the negotiator to match a
>>>>>>> resource twice, because:
>>>>>>>
>>>>>>>A. There's only one negotiator, so it will only match a resource
>>>>>>> once per neogotiaition cycle.
>>>>>>>
>>>>>>>B. It is possible for a resource to be matched twice - if it gets
>>>>>>> matched in one negotiation cycle, but doesn't update the
>>>>>>> collector before the next  negotiation cycle starts. We prevent
>>>>>>> this from happening in two ways:
>>>>>>>  1. The negotiation CANNOT start until at least 20 seconds have
>>>>>>>passed
>>>>>>>since the last one. That usually is good enough.
>>>>>>>  2. The startd can detect if it's been matched twice, and can
>>>>>>>recover,
>>>>>>>if the 20 seconds wasn't good enough.
>>>>>>>
>>>>>>>In Condor-G, because there is no claiming (ie the resource doesn't
>>>>>>> automatically tell the collector that it is no longer able to be
>>>>>>> matched), matching twice is more of a problem. We ususally
>>>>>>> build-in some sort of  claiming when we've deployed matchmaking
>>>>>>> in the past (see the USCMS writeup I pointed you at eariler).
>>>>>>>
>>>>>>>If you advertise your resource to more than one
>>>>>>> collector/negotiator, then matching twice is a real problem. In
>>>>>>> regular condor, it's not possible for a startd to report to two
>>>>>>> different collectors, so it's not a problem.
>>>>>>>
>>>>>>>-Erik
>>>>>>>_______________________________________________
>>>>>>>Condor-users mailing list
>>>>>>>Condor-users@xxxxxxxxxxx
>>>>>>>https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>_______________________________________________
>>>>>>Condor-users mailing list
>>>>>>Condor-users@xxxxxxxxxxx
>>>>>>https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>Condor-users mailing list
>>>>>Condor-users@xxxxxxxxxxx
>>>>>https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>>_______________________________________________
>>>>Condor-users mailing list
>>>>Condor-users@xxxxxxxxxxx
>>>>https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>
>>>>
>>>
>>>
>>>
>
>
>
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users