[Condor-users] SAS on Condor - gotchas with Windows? Linux?

[Chris Thorn <cathorn@xxxxxxxx>]

Hi,

I am working with several economists and statistics grad students on 
both research and production models for producing value-added estimates 
of growth in student learning based on state test score data.

I have been working with the econometricians to try and set up a meeting 
with some applied mathematics folks suggested by the Condor team here at 
UW-Madison. Most of our problems are based on the fact that we are 
dealing with many of the ugly realities of testing in schools - 
retention in grade, mid-year testing, measurement error, bugs in test 
scales that are supposed to be linear but are not, etc. This means that 
both off-the shelf models and software are not up to the task.

We have been using SAS for preprocessing because it is so good at doing 
matrix inversion even when we have multicollinearity problems in the 
data. It's more that the econ folks all know SAS and it does good parlor 
tricks with large matrix operations. All of the heavy lifting for our 
approach occurs in Fortran routines that pick up data output from SAS. 
We are currently experiencing performance problems or actual system 
crashes with 32-bit SAS on Windows. We are afraid that we are going to 
exceed the capabilities of SAS on Linux with our 3-year model. The 
several data sets contain about 500k observations per year.

The problems we are running into indicate to me that we need to look at 
new tool sets and approaches. This sort of thing is not taught in 
econometrics courses. We are doing production work that has both real 
policy and research implications. We are under the gun to make 
improvements and could really use some expert advice.

We need to do lots of preprocessing as well as generate lots of monte 
carlo datasets to test the model under a wide range of possible 
constellations of students classrooms, schools, and districts. That's a 
classic Condor problem.

Most of my folks know Windows best, but are comfortable on Linux. My 
constraint is that while R might be a good option for the long term, I 
am stuck with SAS on either Windows or Linux in the near- to mid-term. I 
don't mind buying licenses to run on UW machines if SAS is really going 
to work.

I see some older traffic on the Condor Users list that suggests that 
Windows SAS does not play well under Condor. I see some help material at 
the Rosen Center that suggests that Linux SAS is no problem at all.

I need to decide whether to buy some licenses and deploy SAS pretty much 
right away. Any advice at all would be very much appreciated.

Chris

--
Dr. Chris Thorn
Asst. Scientist, Value-Added Research Center 
& Director of Technical Services
Wisconsin Center for Education Research
1025 W. Johnson St., Room 767
Madison, WI 53706
http://facstaff.wcer.wisc.edu/cathorn
Tel: 608-263-2709 Fax: 608-265-9300

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature