Re: [HTCondor-users] Condor Win!

["Dunn, George Jr" <dunng@xxxxxxxx>]

Hi Arijit,

 

The lab machines that we are marshaling are windows machines whose primary functions are instruction during the day.

 

I have read where folks were running the R binaries from a network share. For this I just created a silent install script to install R and the necessary packages on each machine.

 

I created a simple .bat file:

(note not sure if the TMP, TEMP, USERPROFILE settings are necessary but I saw it on some matlab examples and figured it could not hurt)

 

set R_SCRIPT="C:\Program Files\R\R-2.15.1\bin\RScript.exe"

set TMP=%CD%

set TEMP=%CD%

set USERPROFILE=%CD%

%R_SCRIPT%  %1 %2 %3 %4  > %2_%3-%4.txt 2>&1

 

The first parameter is the R script itself, the second is the data file the script was using. The third and fourth were parameters associated with the partitioning used to get the runtime down to 12 hours or so.

 

The R script generated another .RData with the output for the job with a similar naming convention as the above std out std err redirection file.

This was done so that there would be uniqueness in the files as 1000’s were being returned to a single directory.

 

Another script was written to generate the appropriate lines in the form of a condor_submit file to queue them all up!

 

Hope this helps!

 

Eddie

 

From: htcondor-users-bounces@xxxxxxxxxxx [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Mukherjee Arijit
Sent: Wednesday, June 26, 2013 2:44 AM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Condor Win!

 

That's great.

Just one quick question - how did you submit the R jobs to Condor? We have done a similar thing (but not in the scale you have mentioned) using /usr/bin/rscript on the cluster nodes.

Regards
Arijit

Arijit Mukherjee
Senior Scientist R&D; Innovation Lab
Tata Consultancy Services
Ph:- 913366367137
Cell:- 9903705285
Mailto: mukherjee.arijit@xxxxxxx
Website: http://www.tcs.com
____________________________________________
Experience certainty.        IT Services
                       Business Solutions
                       Consulting
____________________________________________



From:        "Dunn, George Jr" <dunng@xxxxxxxx>
To:        "htcondor-users@xxxxxxxxxxx" <htcondor-users@xxxxxxxxxxx>,
Date:        06/25/2013 08:31 PM
Subject:        [HTCondor-users] Condor Win!
Sent by:        htcondor-users-bounces@xxxxxxxxxxx

---

Hi All,
 
Just wanted to share a little success story.
 
We are a small public university that has been hit hard in both human and budget resources in recent years (as many schools have). We do have a research computing group but overall our IT is stretched so thin that most all of those folks are on fire duty. A group in our Statistics department was up against a runtime wall trying to run some R jobs in VM’s. There were a total of 8000 jobs that each took 12 hours  to complete (with unfettered access to a single core). By marshaling a few of our labs we were able to run over 1100 simultaneous jobs on 175 ish machines (mostly i7’s a few i5’s) even through several electrical storms that rebooted all the machines and caused all manner of networking errors IN TIME! She is going to add me to the Author list for the paper and we will put a blurb about Condor.
 
I hope this will be a proof of concept for a larger roll-out. I also plan to replace our torque scheduler for ROCKS with this for our mpi jobs.
 
Thank you for such a polished and stable product. I am a fan for life!!!
 
Thanks,
Eddie Dunn
Systems Administrator
Department of Computer Science
University of North Carolina Wilmington_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you