[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Same random value for batches of N jobs within a larger cluster?



Macros on the QUEUE line will expand before the queue line is executed, so  $RANDOM_INTEGER there will randomize only once for all of the jobs.

Arguments = $(RAND_A) $(RAND_B) 
RAND_B = $RANDOM_INTEGER(0,100)
queue 4 RAND_A in ( $RANDOM_INTEGER(0,100), $RANDOM_INTEGER(0,100), $RANDOM_INTEGER(0,100), $RANDOM_INTEGER(0,100) )

This gives output like this:

Args="14 95"
Args="14 5"
Args="14 37"
Args="14 84"
Args="92 67"
Args="92 100"
Args="92 44"
Args="92 94"
Args="43 8"
Args="43 35"
Args="43 67"
Args="43 65"
Args="70 50"
Args="70 86"
Args="70 45"
Args="70 0"

Macros will also expand in the arguments of an include statement, so  If you want RAND_B to also be a set of values that randomizes only once, you can do this:

include command : /bin/echo "RAND_B_LIST=$RANDOM_CHOICE(0,100), $RANDOM_CHOICE(0,100), $RANDOM_CHOICE(0,100),  $RANDOM_CHOICE(0,100)"
Arguments = $(RAND_A) $CHOICE(Step, RAND_B_LIST)
queue 4 RAND_A in $RANDOM_INTEGER(0,100), ...

This can be written more compactly by using a temporary variable for the random expression. 

RV=$RANDOM_INTEGER(0,100)
include command : /bin/echo 'RAND_B_LIST=$(RV), $(RV), $(RV), $(RV)'
Arguments = $(RAND_A) $CHOICE(Step, RAND_B_LIST)
queue 4 RAND_A in ( $(RV), $(RV), $(RV), $(RV) )

The above submit fill will produce 16 jobs, with Arguments that are all possible combinations of 2 sets of 4 random values, like this:

Args="51 55"
Args="51 90"
Args="51 88"
Args="51 6"
Args="47 55"
Args="47 90"
Args="47 88"
Args="47 6"
Args="98 55"
Args="98 90"
Args="98 88"
Args="98 6"
Args="67 55"
Args="67 90"
Args="67 88"
Args="67 6"

-tj

-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Michael Pelletier
Sent: Tuesday, November 14, 2017 8:02 PM
To: HTCondor-Users Mail List (htcondor-users@xxxxxxxxxxx) <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Same random value for batches of N jobs within a larger cluster?

Hi folks,

I've got a situation where I need to iterate over two values in a submit description, and the outer iterator should be tied to the same random number, so that all inner iterator runs have the same random value within each of the outer iterators.

So for A = 1..10 and B = 1..10, A1:B1..10 would have the same random value, A2:B1..10 would have another random value, and so on, out to the ten batches of ten tasks, for a total of ten random values.

The trouble is that $RANDOM_INTEGER() updates for each job, but batches of 10 jobs in the example above would need to share the same value.

I'm puzzling over whether there's some clever way to do this entirely in the submit description, ideally with varying numbers of outer iterators.

Maybe I'd just need to have an include command generate a scratch file with an appropriate-length list of random numbers from Perl or the shell or whatever, and "queue 10 from" that file?


Michael V. Pelletier
Principal Engineer
Information Technology
Future Technologies & Cloud
Integrated Defense Systems
Raytheon Company
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/