Re: [Condor-users] cluster number in shell script (sh)

["David J. Herzfeld" <david.herzfeld@xxxxxxxxxxxxx>]

Alexandru:

You may want to consider defining a custom key-value pair in your
submission file that you can keep track of. It looks like you are
already using the variable nn to keep track of a unique "cluster key,"
so it seems like a good choice.

Here is a very quick-n-dirty example of what I mean (obviously it will
need a bit of tweaking for your application). In the example, I use
MY_CLUSTER_VALUE to store an integer number (which you could replace
with ${nn} in your script). I then just look for all log files that
start with this cluster value after waiting on the log file.

#!/bin/bash

# The log directory
LOG_DIR=logs
rm -rf $LOG_DIR 2>/dev/null
mkdir $LOG_DIR

# Submission files (that we are making)
SUBMIT_FILE=example.sub

# Create the submission file
cat << EOF > $SUBMIT_FILE
Universe = vanilla
Executable = /bin/sleep
Arguments = 10
MY_CLUSTER_VALUE = 1
Log = ${LOG_DIR}/log
Output = ${LOG_DIR}/\$(MY_CLUSTER_VALUE).\$(Process).out
Error = ${LOG_DIR}/\$(MY_CLUSTER_VALUE).\$(Process).err
when_to_transfer_output = ON_EXIT
transfer_executable = NO
queue 10

MY_CLUSTER_VALUE = 2
queue 10
EOF

condor_submit ${SUBMIT_FILE}

condor_wait ${LOG_DIR}/log

# This could easily be scripted...
cat $LOG_DIR/1.*.out $LOG_DIR/1.*.err > $LOG_DIR/1.log
rm -rf $LOG_DIR/1.*.out $LOG_DIR/1.*.err
cat $LOG_DIR/2.*.out $LOG_DIR/2.*.err > $LOG_DIR/2.log
rm -rf $LOG_DIR/2.*.out $LOG_DIR/2.*.err
exit 0

Best regards,
David


On 11/30/2010 06:57 AM, Alexandru Chiscan wrote:
> On 11/29/2010 05:13 PM, Steven Timm wrote:
>> yes, set
>> log=filename.$(cluster)
> Yes, I know about $(cluster). The problem is that I want to know the
> cluster number generated by a condor_submit command called in a shell
> script like this:
> 
> #!/bin/sh -f
> job=$0.$$.submit
> LOGDIR=$0.$$
> mkdir $LOGDIR
> 
> arg="-v"
> echo "Universe=vanilla" >> ${job}
> echo "Executable=/bin/exe" >> ${job}
> echo "Input=/dev/null" >> ${job}
> echo "Output=${LOGDIR}/out.\$(Process) " >> ${job}
> echo "Error=${LOGDIR}/err.\$(Process) " >> ${job}
> echo "log=${LOGDIR}/log " >> ${job}
> echo "transfer_executable=yes" >> ${job}
> echo "when_to_transfer_output = ON_EXIT" >> ${job}
> 
> nn=0
> for ((j=1;j<1000;j++)) ; do
>     echo "Arguments=${arg} $j" >> ${job}
>     echo "Queue" >> ${job}
>     echo "" >> ${job}
>     nn=`expr $nn + 1`
> done
> 
> condor_submit ${job}
> 
> condor_wait ${LOGDIR}/log
> 
> Here I want to know the cluster number in order to regroup all the
> Output and Error files in one file named $(cluster).log
> For the moment the only ideea I have is to parse the output of
> condor_submit like this:
> 
> c_submit=`condor_submit ${submit}`
> exit_status=$?
> if [ ${exit_status} != 0 ]; then
>     exit ${exit_status}
> fi
> *cluster=`echo ${c_submit} | grep "submitted to cluster" | awk '{print
> $NF}' `*
> 
> but this is dependent on the output of the condor_submit command which
> could change in other version of Condor.
> 
> I want to use the cluster number because it is unique (unlike $$ which
> is the PID of the shell and for sure will be reused by the system).
> 
> Best regards,
> Alexandru
> 
>>
>> in your condor submit file
>> and the log file will be created accordingly.
>>
>> Steve timm
>>
>>
>> On Mon, 29 Nov 2010, Alexandru Chiscan wrote:
>>
>>>    Hello all,
>>>
>>> I am using shell scripts to generate condor submit files and to submit
>>> them.
>>> The scripts generate clusters with many jobs (hundreds) and I want at
>>> the end to have in one log file the output and the error log of all the
>>> jobs in the cluster. For the moment I am using the shell name and pid
>>> for the log name ( $0.$$ ) but the pid will be reused and I will like to
>>> use the cluster number for it - which should be unique (or not?).
>>>
>>> Is there any way to get the cluster number in the script without having
>>> to parse the output of the condor_submit command?
>>>
>>> cluster=`condor_submit job.submit | grep "submitted to cluster" | awk
>>> '{print $6} `
>>> echo "cluster $cluster"
>>>
>>> Regards,
>>> Alexandru
>>>
>>> _______________________________________________
>>> Condor-users mailing list
>>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
>>> with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/condor-users/
>>>
>>
> 
>