[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Condor-users] Reading local files on a node

Hello all, we have a bit of a dilemma.

We are running jobs that have extremely large input files, and the current method is to read all of it into memory at the start - but this is causing machines to hang due to using too much memory. The other method is to stream the data from the input file on the frontend to the compute node - but this is extremely slow.

Is there a way to automatically transfer and store the input data file on the compute node using the submit file? So that it can be read by the program.
It has to be vanilla universe.