Re: [HTCondor-users] quotes around arguments for jobs sent to the grid environment

[Patty J Bragger-Wilkinson <bragger_patty_j@xxxxxxxxx>]

In case anyone stumbles across this thread, here is the solution I've implemented

First, we use remote_cerequirements and pass along a flag that will set an env var that holds a single quote, and also sending along a job name that will help us tie together the jobs in condor and the jobs in grid engine (That part isn't actually necessary for the problem outlined, but may be helpful for someone else.)

+remote_cerequirements = "set_squote == 1 && job_name == \"condor_cluster_$(Cluster)\"  "

Then we are using the "old-style" for arguments in the submit file:

arguments = -T something -F \"BQ=20;MQ=20;NR=5\" -q 'I\${squote}m'\" sick of quotes\"  -c /some/path

Then in the sge_local_submit_attributes.sh file, we have added the following logic:

if [[ -n $job_name ]]; then
  echo "#$ -N $job_name"
fi
if [[ -n $set_squote ]]; then
  echo "export squote=\"'\" "
fi

Then say say the executable is something that just does this:
echo $1
echo $2
...

The resultant bl_* file that gets generated by blahp/gahp will contain the following lines.  I've stripped out a bunch for the sake of simplicity

#$ -N "condor_cluster_6769"
export squote="'" 

# Command to execute:
/home/rx97629/bin/echo_arguments.sh -T something -F "BQ=20;MQ=20;NR=5" -q I${squote}m" sick of quotes" -c /some/path < "/dev/null" > "/my/working/directory/test_quotes.out.6769.0" 2> "/my/working/directory/test_quotes.err.6769.0" &

__END__

This results in an output file test_quotes.out.6769.0 that contains the following:
-T
something
-F
BQ=20;MQ=20;NR=5
-q
I'm sick of quotes
-c
/some/path


The only downside to this approach is that the argument definition in the submit file is not compatible across the environments.  This change has to be applied only if submitting to a grid universe, but the syntax must be different when submitting to the vanilla or other universes.

Thanks,
Patty



> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On
> Behalf Of Patty J Bragger-Wilkinson
> Sent: Thursday, April 23, 2015 7:35 PM
> To: htcondor-users@xxxxxxxxxxx
> Subject: [HTCondor-users] quotes around arguments for jobs sent to the grid
> environment
> 
> We have a workflow that currently runs fine in the vanilla universe using the
> following syntax for itâs arguments:
> 
> Executable = /some/path/to/executable.pl Arguments = -T something âF
> âa=1;b=2;c=3â âc /some/path
> 
> Iâm trying to move things to a grid universe and Iâm having trouble getting
> these quotes to propagate down the chain to the submit file that gets
> generated for the Grid Engine scheduler.  The command that gets populated
> in that submit script looks like this:
> 
> /some/path/to/executable.pl âT something âF a=1;b=2;c=3 âc /some/path
> ...
> 
> Followed by other things that take care of redirecting the inputs and outputs
> etc..
> 
> The single quotes are getting stripped off, which makes the submit file
> effectively invalid.   I have tried using the new-syntax for arguments as well,
> but that has a rather bizarre effect:
> 
> Arguments = â-T something âF âa=1;b=2;c=3â âc /some/pathâ
> 
> Puts this in the GE submit file:
> /some/path/to/executable.pl ââT\â \âsomething\â \ââF\â \âa=1;b=2;c=3\â
> \ââc\â \â/some/pathâ ...
> 
> So the single quotes seem to be stripped off, then all of the args are passed
> in one massive double quoted string, and lots of superfluous quotes have
> been added to it.
> 
> Iâve also yet to figure out how to pass an argument that may contain a single
> quote (rather than surrounded by single quotes.)
> 
> Has anyone had any experience with something like this?  Could you point
> me to something that may help?  My next option from here is to try to hack
> the sge_local_submit_attributes.sh script and pass something in through
> +remote_cerequirements that will set an env var to represent a single quote
> so that it will be interpreted only at execute time.. but that seems like a lot of
> effort to go through to use a single quote in an argument, and it feels like
> there should be an easier path that Iâm just missing.
> 
> Any help would be appreciated.
> 
> Thanks!
> Patty
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/