[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Same random value for batches of N jobs within a larger cluster?

On Wed, 2017-11-15 at 02:01:33 +0000, Michael Pelletier wrote:
> Hi folks,
> I've got a situation where I need to iterate over two values in a submit description, and the outer iterator should be tied to the same random number, so that all inner iterator runs have the same random value within each of the outer iterators.
> So for A = 1..10 and B = 1..10, A1:B1..10 would have the same random value, A2:B1..10 would have another random value, and so on, out to the ten batches of ten tasks, for a total of ten random values.
> The trouble is that $RANDOM_INTEGER() updates for each job, but batches of 10 jobs in the example above would need to share the same value.
> I'm puzzling over whether there's some clever way to do this entirely in the submit description, ideally with varying numbers of outer iterators.
> Maybe I'd just need to have an include command generate a scratch file with an appropriate-length list of random numbers from Perl or the shell or whatever, and "queue 10 from" that file?

Wouldn't a two-level DAG do the trick? First-level jobs create random numbers,
delegate them to the next level? This probably wouldn't be possible within the
submit context, but there are files...

Just a quick thought,