[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[HTCondor-users] condor as R parallel backend

R has a variety of ways to run jobs in parallel, many of which can now automatically dispatch jobs to a slurm, torque, sge, etc. cluster. All that's required to add condor to this list is to create a template file of the same sort used for these other backends, but, after trying, it seems outside my current skill set. The two most promising avenues, already with filed issues, are:


GitHub: https://github.com/mschubert/clustermq
Issue: https://github.com/mschubert/clustermq/issues/74
Example SGE template: https://github.com/mschubert/clustermq/blob/master/inst/SGE.tmpl

future.batchtools (which wraps the batchtools package)

GitHub: https://github.com/HenrikBengtsson/future.batchtools
Issue https://github.com/HenrikBengtsson/future.batchtools/issues/29
Custom template stub: https://github.com/HenrikBengtsson/future.batchtools/blob/develop/R/batchtools_custom.R

[another potential solution with future.batchtools is to add the functionality directly to batchtools; relevant SGE template: https://github.com/mllg/batchtools/blob/master/inst/templates/sge-simple.tmpl ]

These packages are under active development and are being integrated into other software (i.e., as back ends for the drake workflow package https://ropenscilabs.github.io/drake-manual/hpc.html#parallel-backends). I imagine someone on this mailing list has the expertise to just change a few things in the template to make it work for condor (moving files could be an issue, though a solution that relies on file syncing by some service outside condor would be fine).

Can anyone help?

Thank you,

Brian O'Meara, http://www.brianomeara.info, especially Calendar, CV, and Feedback
Associate Professor, Dept. of Ecology & Evolutionary Biology, UT Knoxville
Associate Head, Dept. of Ecology & Evolutionary Biology, UT Knoxville