Re: [HTCondor-users] Apache Spark on HTCondor

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Canât say Iâve ever tried it!

Thereâs an interesting project to be had here: what you really would like is to have HTCondor maintain a certain number of Spark workers (as opposed to submitting jobs to HTCondor).  Seems like itâs an ideal candidate for the job factory work being done...

Brian

On Aug 2, 2016, at 8:23 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

Dear experts,

I am currently trying to figure out what is needed in order to run Apache Spark [1] on an HTCondor cluster [2]. It seems that Spark can use external scheduling (YARN, Mesos), which means that at least in theory, this should be possible.

Before I dive too deep into Spark, I wanted to ask around if someone has tried this before.

There have been talks about Spark at the last HTCondor Week [3], so it seems that there is interest.

Cheers,

Luke

[1]
http://spark.apache.org/

[2]

Our cluster is Hadoop (HDFS + YARN) but with YARN disabled - we use HTCondor instead for scheduling (similar to some US sites?)

[3]
http://research.cs.wisc.edu/htcondor//HTCondorWeek2016/presentations/WedBenton_Dist2Data.pdf
--

*********************************************************
  Dr Lukasz Kreczko           
  Research Associate

  Department of Physics

  Particle Physics Group

  University of Bristol
  HH Wills Physics Lab
  University of Bristol
  Tyndall Avenue
  Bristol
  BS8 1TL

  +44 (0)117 928 8724 

  L.Kreczko@xxxxxxxxxxxxx

  

  A top 5 UK university with leading employers (2015)

  A top 5 UK university for research (2014 REF)

  A world top 40 university (QS Ranking 2015)

*********************************************************

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/