Re: [HTCondor-users] Apache Spark on HTCondor

[L Kreczko <L.Kreczko@xxxxxxxxxxxxx>]

Hi Brian,

On 3 August 2016 at 04:17, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

Canât say Iâve ever tried it!

Thereâs an interesting project to be had here: what you really would like is to have HTCondor maintain a certain number of Spark workers (as opposed to submitting jobs to HTCondor).

But is that not equivalent to wasting resources? What Spark does seems very similar to DAGMan: a control process (Driver program) with a few worker node processes [1]. In the given link the specifications for a cluster setup are also given. But it seems that one would have to implement a cluster manager for HTCondor within Spark.

ÂSeems like itâs an ideal candidate for the job factory work being done...

In that scenario I assume you would run Spark in stand-alone in the few reserved slots and scale out when someone starts a Spark application?

This is probably the easiest to implement (compared to the cluster manager).

So, when do we start? ;)

BTW: Please take my writing with a grain of salt, still have lots to learn about both HTCondor and even more about Spark.

Cheers,

Luke

Brian

[1]
http://spark.apache.org/docs/latest/cluster-overview.html

On Aug 2, 2016, at 8:23 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

Dear experts,

I am currently trying to figure out what is needed in order to run Apache Spark [1] on an HTCondor cluster [2]. It seems that Spark can use external scheduling (YARN, Mesos), which means that at least in theory, this should be possible.

Before I dive too deep into Spark, I wanted to ask around if someone has tried this before.

There have been talks about Spark at the last HTCondor Week [3], so it seems that there is interest.

Cheers,

Luke

[1]
http://spark.apache.org/

[2]

Our cluster is Hadoop (HDFS + YARN) but with YARN disabled - we use HTCondor instead for scheduling (similar to some US sites?)

[3]
http://research.cs.wisc.edu/htcondor//HTCondorWeek2016/presentations/WedBenton_Dist2Data.pdf
--

*********************************************************
 Dr Lukasz Kreczko     ÂÂ
 Research Associate

 Department of Physics

 Particle Physics Group

 University of Bristol
 HH Wills Physics Lab
 University of Bristol
 Tyndall Avenue
 Bristol
 BS8 1TL

 +44 (0)117 928 8724Â

 L.Kreczko@xxxxxxxxxxxxx

ÂÂ

 A top 5 UK university with leading employers (2015)

 A top 5 UK university for research (2014 REF)

 A world top 40 university (QS Ranking 2015)

*********************************************************

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/