[HTCondor-users] Apache Spark on HTCondor

[L Kreczko <L.Kreczko@xxxxxxxxxxxxx>]

Dear experts,

I am currently trying to figure out what is needed in order to run Apache Spark [1] on an HTCondor cluster [2]. It seems that Spark can use external scheduling (YARN, Mesos), which means that at least in theory, this should be possible.

Before I dive too deep into Spark, I wanted to ask around if someone has tried this before.

There have been talks about Spark at the last HTCondor Week [3], so it seems that there is interest.

Cheers,

Luke

[1]
http://spark.apache.org/

[2]

Our cluster is Hadoop (HDFS + YARN) but with YARN disabled - we use HTCondor instead for scheduling (similar to some US sites?)

[3]
http://research.cs.wisc.edu/htcondor//HTCondorWeek2016/presentations/WedBenton_Dist2Data.pdf
--

*********************************************************
 Dr Lukasz Kreczko     ÂÂ
 Research Associate

 Department of Physics

 Particle Physics Group

 University of Bristol
 HH Wills Physics Lab
 University of Bristol
 Tyndall Avenue
 Bristol
 BS8 1TL

 +44 (0)117 928 8724Â

 L.Kreczko@xxxxxxxxxxxxx

ÂÂ

 A top 5 UK university with leading employers (2015)

 A top 5 UK university for research (2014 REF)

 A world top 40 university (QS Ranking 2015)

*********************************************************