Re: [HTCondor-users] Process list of files

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Hi Roger,

If I'm reading your email right, I think you want to look at the "-append" argument to condor_submit.  It would look something like this:

for s in $( ls <some file pattern); do
  <some setup>
  condor_submit my_submit_file -append my_file=$s
done

Here's the man page for "-append":

"""
       -append command

          Augment  the  commands in the submit description file with the given command. This command will be considered to immediately precede the Queue command within the submit description
          file, and come after all other previous commands. The submit description file is not modified. Multiple commands are specified by using the -append option multiple times. Each  new
          command is given in a separate -append option. Commands with spaces in them will need to be enclosed in double quote marks.
"""

You would then have the following arguments and executable line:

executable=<some command>
arguments="<my arguments> $(my_file)"

The submit file is a macro language; the "arguments" command will have $(my_file) expanded from what you put in the CLI.

Hope this is what you're looking for,

Brian

> On Dec 5, 2014, at 7:52 PM, Roger Herikstad <roger.herikstad@xxxxxxxxx> wrote:
> 
> Hi everyone, 
> I recently started testing HTCondor as a replacement for our ageing SGE setup. Our lab uses Mac Pro's as combined compute nodes/workstations, and though SGE works reasonable well with this setup, the fact that it is no longer open source (and has a somewhat convoluted build/install process), made me look around for alternatives. I've managed to get HTCondor up and running with few of our nodes, and so far I really like what I'm seeing. I have one question, though. Our typical SGE workflow would be something like this:
> 
> From the terminal:
> 
> for s in $( ls <some file pattern> ); do <some setup>; echo "cd $PWD; <some command with arguments> $s" | qsub <sge arguments>; done
> 
> In other words, we would have a list of files containing data to be processed, and we would send of one SGE job per file. What is the best way to achieve this workflow under HTCondor? I came across this: 
> https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=VaryArgumentsByProcId
> which gets me part of the way, but if I understand it correctly, I'd have to manually define the list of files as a ClassAd in the submit file. This is certainly doable, but knowing my users, the more steps you have go through to submit jobs, the less likely they are to use the system. 
> i'd appreciate any thoughts on this from the list. Thanks!
> 
> 
> Roger Herikstad
> Research Fellow
> SiNAPSE
> National University of Singapore
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/