Re: [Condor-users] Sample parameter sweep application

["Matias Alberto Gavinowich" <mattlistas@xxxxxxxxx>]

Hello Ian,

Thank you for the information you provided. I've been studying the
suggestions I gathered and also playing with a couple of apps of my
own, so now I'll briefly share my experience on what you described.

First, let me clarify what went on with the original posting: my
intention was to gather several samples that I could look at, compare
with my own, evaluate, and eventually choose from. That is, it was
more of an information collection request rather than a call for help
on a current problem. This is why I sent the email to more than one
list, since every answer did count, and indeed I got useful replies
from more than one list. Also, I sent it to a small subset of the
lists I am subscribed to - those I felt the posting applied to. In the
rare occasions in which I may have cross-posted, I did so for this
reason or because I was making a release announcement. In any case, I
apologize for the inconvenience of having gotten my email more than
one time.

Now, to the main point of this email. I played around with MOLREP, and
could produce a couple of plots. I am not sure if I will actually end
up including MOLREP in the demos because the executable is larger than
what I hoped for. I know that in a situation - more common than mine -
in which the execution hosts are close or even have a shared file
system, the size of the executable is not an issue. However, my
particular test bed has no shared file system, need transferring of
the executable since it's not in every host, and is limited to roughly
32 KB/s upload bandwidth. So there I have a couple of minutes of
transfer time - and the transfer sometimes does fail. Ideally, I
wanted to show that running each job on a separate computer increased
overall time, but if the job takes only seconds and when I execute
them remotely I have to add a couple of mins to each one, then it
still 'seems' quicker to run the jobs locally in a serial way.

I am also not sure if this overhead is once per job or once per
execution host, since I am sending all jobs in the same cluster and
the Condor Manual states that this allows Condor to manage resources
more efficiently, although I was not able to find specifically what
this refers to and if the transfered executable would be cached on an
execution host for the next matched job belonging to the same cluster,
if any. I may ask this in a separate thread since it deviates a bit
from the issue of this one. If the executable is cached I might be
able to demo larger executables, since most times I get only 2 or 3
free hosts for a 10 to 100 jobs cluster, so if the executable was
transfered only 2 or 3 times instead of 10-100 I may be able to
neglect the transfer time.

I am also currently doing one-parameter variation simulations, though
I had considered varying several as you mention (pdb, RESMAX, SIM,
COMPL would be four).

Anyway, about MOLREP itself. I downloaded the binary linux version
from the link you provided (I had some issues compiling the source). I
found that there was a new invocation style, which I used. I also got
the test MTZs and PDBs from the molrep_check/data directory in the
source distribution. After downloading the extra files provided on the
page (for CCP4) and configuring the CLIB environmental variable, I
could get .ps plots when running the executable just with the MTZ, in
this way: ./molrep_linux -f test.mtz (the TGase.mtz file also worked).
However, when trying to use a pdb (with the -m switch) I did not get
any plot, although some output files were generated. As for the PDBs,
I could only get the ones in molrep_check/data to work. The ones in
the link from your email were not recognized by the program, so I
might have missed something there, but from my studying of the molrep
code there is a fragment in molrep_subr.f, CHECK_COORD_FILE
subroutine:

        IF(LINE(1:6).EQ.'ATOM  '.OR.LINE(1:6).EQ.'HETATM') THEN
          PDB = 'Y'

which looks for ATOM or HETATM, which are present in the PDBs in
molrep_check/data but not in the ones I downloaded from the link.
Perhaps there is some file conversion I overlooked.

In summary, I would be able to do a parameter sweep varying the MTZ,
although that may or may not make real-world sense, I am not sure of
that. Incorporating a PDB into the picture produced no .ps in my
tests. I tested also with the commands in your email (by means of the
k switch in molrep) with the same luck.

Cheers,

Matias


On Mon, Dec 8, 2008 at 7:32 PM, Ian Stokes-Rees
<ijstokes@xxxxxxxxxxxxxxxxxxx> wrote:
> Matias,
>
> I stuck my foot in my mouth there, so to make up for it, let me help you
> however I can.  I have a nice and relevant application which I am running
> right now, and which will give you some good graphics as output.  It does
> protein model molecular replacement, and there are a number of different
> parameters that can be varied through a command file.  You need to download
> and compile the application from here:
>
> http://www.ysbl.york.ac.uk/~alexei/molrep.html#installation
>
> then I can give you a set of data files or you can wget them from here:
>
> http://abitibi.sbgrid.org/se/data/shared/biodb/balbes/
>
> and you need one "MTZ" file, for example:
>
> http://abitibi.sbgrid.org/~ijstokes/mr-jobs/tgase/0-input/TGase.mtz
>
> Then you want to run the following loops:
>
> 1. Loop over every file in the "biodb/balbes" directory.
>
> 2. For each file, use the base input parameter file "molrep.cmd":
>
> _DOC Y
> LABIN F=FP SIGF=SIGFP
> _SCORE N
> _FUN A
> _RESMAX 3.0
> _NMON 1
> _MODE F
> _PACK Y
> _ANISO Y
> _SURF N
> _SIM 1.0
> _COMPL 0.5
> _NP     10
> _NPT    10
> _END
>
> 3. you execute the application by the command:
>
> cat molrep.cmd | molrep HKLIN TGase.mtz MODEL 2oqo.pdb
>
> 4. then repeat for the following variations:
> and vary _RESMAX in intervals of 1.5 starting at 8.0 and lowering to 3.5
> and vary _SIM in intervals of 0.1 from 0.0 to 1.0
> and vary _COMPL in intervals of 0.1 from 0.0 to 1.0
>
> Let me know if you need any more suggestions on how to proceed with this.
> I'd be very interested to see your results!  Each invocation should take
> 10-30 seconds, so you may want to have a more coarse grid and user fewer
> input files.
>
> Cheers,
>
> Ian
>
> Matias Alberto Gavinowich wrote:
>
> Hello,
>
> I am working on a software capable of handling parameter sweep
> application submissions on a cluster or grid environment (for the time
> being, I am using Condor), for my thesis. I am looking for sample
> applications which I may use to test the software I've developed. I
> would appreciate your help and suggestions, perhaps someone in the
> community can point me to some application which meets my needs.
>
> What I am looking for is a non-data intensive application for which it
> may be desired to perform several runs varying a parameter. This
> parameter is an input file. For instance, something like a program
> which finds the roots of a mathematical function, which I would use
> as:
>
> find_roots range.txt
>
> Range would contain something like 1-10000, and I would submit several
> ranges, so each run can take place in a separate computer and while
> one processes a range, the other processes another range.
>
> The program can produce output on the standard output as well as on
> another file on disk, or both. The key is that both find_roots and
> range.txt are files of relatively small size, and that the benefit of
> running each range on a separate computer is that overall speed is
> increased. For demonstration purposes, this program should take a bit
> to run, but I would like to be able to make a couple of runs fit into
> a short demo. It's purpose and produced output should also be
> relatively easy to understand by an audience new to it.
>
> I have been looking for real-world (or near real-world) applications
> that resemble what I've just described, but the ones I found require
> large databases to be present, require several input files instead of
> one, the files involved are very large, etc.
>
> If anyone can point me to an application which works like I described,
> I'd greatly appreciate it. Availability of source code is a plus, but
> not essential.
>
> Thanks!
>
> Matt
>
>
> --
> Ian Stokes-Rees                            W: http://sbgrid.org
> ijstokes@xxxxxxxxxxxxxxxxxxx               T: +1 617 418-4168
> SBGrid, Harvard Medical School             F: +1 617 432-5600
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
>