[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Apache Spark on HTCondor

Canât say Iâve ever tried it!

Thereâs an interesting project to be had here: what you really would like is to have HTCondor maintain a certain number of Spark workers (as opposed to submitting jobs to HTCondor).  Seems like itâs an ideal candidate for the job factory work being done...


On Aug 2, 2016, at 8:23 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

Dear experts,

I am currently trying to figure out what is needed in order to run Apache Spark [1] on an HTCondor cluster [2]. It seems that Spark can use external scheduling (YARN, Mesos), which means that at least in theory, this should be possible.

Before I dive too deep into Spark, I wanted to ask around if someone has tried this before.
There have been talks about Spark at the last HTCondor Week [3], so it seems that there is interest.



Our cluster is Hadoop (HDFS + YARN) but with YARN disabled - we use HTCondor instead for scheduling (similar to some US sites?)

  Dr Lukasz Kreczko           
  Research Associate
  Department of Physics
  Particle Physics Group

  University of Bristol

  HH Wills Physics Lab
  University of Bristol
  Tyndall Avenue
  BS8 1TL

  +44 (0)117 928 8724 
  A top 5 UK university with leading employers (2015)
  A top 5 UK university for research (2014 REF)
  A world top 40 university (QS Ranking 2015)
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: