[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Different arguments file for each job

I just finish installing condor and I'm trying to figure out a way to have submit a job with different file as argument for each job.
I have a directory with 100s of files, for each file, I run the following program (simplified call):
blast -i <one_file>

Your submit file is correct and it's the only way to do it--Condor will not look through your directory at each file.

Are you on a Unix system? If so, all you need to do is write simple script that will generate a submit file for you. I've written such a script and you can try it out. It will generate one big submit file for each .dat file in a directory. It looks like this:


echo Creating submit file...
cat > submit_all_jobs <<EOF
# Here we have the information shared between all jobs
executable              = blastall
Universe                = vanilla
Error                   = err.\$(Process)
Output                  = out.\$(Process)
Log                     = blast.log
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT
Requirements            = Memory > 200

# Here is each job

for dat in `ls *.dat`
cat >> submit_all_jobs <<EOF
arguments               = -i ${date}
transfer_input_files    = ${dat}
queue 1


echo Done. Submit file is  called submit_all_jobs.

Remove the lines with hyphens before placing this into a file in your directory with all of your .dat files. Call the file something like make-submit. Then do "chmod +x make-submit". Then just run make-submit, and you will have a submit file that will submit a job for each dat file. It will generate a cluster of jobs. That is, each job will have the same cluster number and a different process number. For example, your jobs may all be in cluster 10, so you'll see your jobs listed as 10.0, 10.1, 10.2, and so on.

I hope this helps.