Re: [Condor-users] Condor deployment on multiple distributed resources

[Gabriel Mateescu <gabriel.mateescu@xxxxxxxxx>]

Sergio,

Condor has the three key components
of a workload management system:

1. Resource information (Condor collector
    and start daemons);

2. Job management (Condor sched daemon
    and adapters for communicating with remote
    resources);

3. Job scheduling (Condor negotiator that uses
    matchmaking between the resource classAds
    and job classAds).

The condor sched daemon takes care of
jobs submitted to that sched (you can have
scheds running on multuple server for
scalability); it works with the negotiator to
select a resource (which can be a server or
a remote cluster managed by a separate workload
manager such as PBS) that matches the job.

The sched daemon communicates with the
remote resource using a resource-specific
adapter, e.g., a GT5, LSF, or PBS adapter.
The adapter is co-located with the scheduler
and communicates over the network with the
remote resource.

On the remote cluster you need to
install the "information provider" that
will give the collector daemon of
your Condor installation information
about the resources provided by the
cluster.

You also need for your Condor installation
to have the right adapter between
sched and the remote cluster.

Gabriel


On Wed, Jun 6, 2012 at 11:30 AM,  <sergio.maffioletti@xxxxxxxxxx> wrote:
> Hi Derek
>
> thanks for you reply
> too experimental for my taste :)
>
> let's ease the constrains then
> what would be a possible deployment scenario if we would be allowed to
> deploy some condor stuff on the remote clusters frontend ?
>
> what condor component should be deployed on the site to integrate the
> underlying batch system ?
> what condor component would act as scheduler to dispatch jobs to the remote
> sites ? (e.g. CondorG ?)
> again
> what authentication model would/should be used ?
>
> I apologise if some (if not all) of these question makes little sense,
> but not having previous experience with condor, it quite difficult to
> understand the relation between the different components
>
> thanks
>
>
> Sergio :)
>
> sergio.maffioletti@xxxxxxxxxx
> GC3: Grid Computing Competence Center
> http://www.gc3.uzh.ch/
> University of Zurich
> Winterthurerstrasse 190
> CH-8057 Zurich Switzerland
> Tel: +41 44 635 4222
> Fax: +41 44 635 6888
>
>
> -----condor-users-bounces@xxxxxxxxxxx ha scritto: -----
> Per: Condor-Users Mail List <condor-users@xxxxxxxxxxx>
> Da: Derek Weitzel
> Inviato da: condor-users-bounces@xxxxxxxxxxx
> Data: 06/06/2012 04.45PM
> Oggetto: Re: [Condor-users] Condor deployment on multiple distributed
> resources
>
>
> Hi Sergio,
>
> There has been recent developments to help with your use case.  Specifically
> to not run anything on the remote cluster, and to control everything from
> the client side.  This is the BLAHP over SSH work:
> https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2421
>
> This BLAHP over SSH allows you to submit jobs through the SSH interface to a
> cluster.  No middleware needed on the remote clusters.
>
> Additionally, we have built software on top of the BLAHP over SSH to create
> a condor overlay on the different resources, called BOSCO (BLAHP over SSH
> Condor Overlay).  But this is Beta, at best.
>
> If you're interested in trying this software, drop me a line.
>
> If all you have is a hammer...
> Derek Weitzel
> Graduate Research Assistant
> University of Nebraska Holland Computing Center
>
> On Jun 5, 2012, at 6:15 AM, sergio.maffioletti@xxxxxxxxxx wrote:
>
>> Hi
>>
>> probably this subject has been already discussed previously, but being new
>> to the mailing list, I take the liberty of asking again
>>
>> we're getting 'cold feet' on the grid technologies and we'd like to
>> explore alternative possibilities for accessing distributed resources
>> in a manner that allow us to reduce the operational overhead footprint at
>> the individual sites and, at the same time, that would give us also access
>> to private/public cloud infrastructures as a mean to expand the resource
>> portfolio
>>
>> in this context, we'd like to understand whether and how condor could be
>> used as a central component to achieve this level of integration/aggregation
>>
>> more specifically:
>> - how condor could/should be deployed in order to aggregate various
>> distributed computing clusters each of them controlled by a different
>> incarnation of a batch system (SGE, LSF, PBS, Slurm)
>> - what components needs to be deployed ? (please do not tell me we need to
>> deploy globus on each site... :)
>> - is it possible to avoid deploy anything at the individual site level ?
>> - what would be the authentication mechanism in this case ?
>> - how private/public cloud resources could be integrated ?
>> - how the whole thing would be used from the client side ?
>>
>> the usage idea would be to develop application-specific tools on top of
>> such infrastructure, thus it it also very important to minimise the number
>> of component deployed (if possible)
>>
>> thanks for any hint
>>
>> Cheers
>> Sergio :)
>>
>> sergio.maffioletti@xxxxxxxxxx
>> GC3: Grid Computing Competence Center
>> http://www.gc3.uzh.ch/
>> University of Zurich
>> Winterthurerstrasse 190
>> CH-8057 Zurich Switzerland
>> Tel: +41 44 635 4222
>> Fax: +41 44 635 6888
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/condor-users/
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>