[HTCondor-users] Prevent slot from re-entering pool after Shadow exception/cannot allocate memory

["Zell, Wesley" <wzell@xxxxxxxx>]

I am using Condor on a dedicated CentOS7 cluster to run a parameter estimation process (PEST/BeoPEST). In brief: a master instance of the software runs on the submit node, while worker instances of the software run within the Condor working directory on the execute nodes.

On occasion, after the process hums along happily for a few hours, I get the following combination of errors that crashes the job:

1. The communication between the submit node and an execute node is interrupted;

2. When the communication is re-established (or appears to be), the software instance on the submit node crashes.

My specific questions:

a. What is the cause of the first error (i.e., the communication interruption - see log file excerpt below)?

b. If the communication disruption cannot be prevented, can I prevent the slot from rejoining the pool after interruption? (My software has some resiliency with respect to failed workers, but apparently not to workers that disappear then rejoin.)

Thanks.

Here are more details:

***************************

(1) Excerpt from log file:

***************************

...

006 (113.032.000) 03/07 23:57:30 Image size of job updated: 2695712

    445 Â- ÂMemoryUsage of job (MB)

    2010192 Â- ÂResidentSetSize of job (KB)

...

022 (113.045.000) 03/07 23:57:31 Job disconnected, attempting to reconnect

  Socket between submit and execute hosts closed unexpectedly

  Trying to reconnect to slot6@node3 <192.168.1.30:32998?addrs=192.168.1.30-32998>

...

023 (113.045.000) 03/07 23:57:31 Job reconnected to slot6@node3

  startd address: <192.168.1.30:32998?addrs=192.168.1.30-32998>

  starter address: <192.168.1.30:7920?addrs=192.168.1.30-7920>

...

007 (113.045.000) 03/07 23:57:31 Shadow exception!

    Erroage of job (MB)

    2010192 Â- ÂResidentSetSize of job (KB)

...

022 (113.045.000) 03/07 23:57:31 Job disconnected, attempting to reconnect

  Socket between submit and execute hosts closed unexpectedly

  Trying to reconnect to slot6@node3 <192.168.1.30:32998?addrs=192.168.1.30-32998>

...

023 (113.045.000) 03/07 23:57:31 Job reconnected to slot6@node3

  startd address: <192.168.1.30:32998?addrs=192.168.1.30-32998>

  starter address: <192.168.1.30:7920?addrs=192.168.1.30-7920>

...

007 (113.045.000) 03/07 23:57:31 Shadow exception!

    Error from slot6@node3: Couldn't reopen Logs/worker_113_45.out to stream stdout: Cannot allocate memory

    0 Â- ÂRun Bytes Sent By Job

    165274896 Â- ÂRun Bytes Received By Job

Âfrom slot6@node3: Couldn't reopen Logs/worker_113_45.out to stream stdout: Cannot allocate memory

    0 Â- ÂRun Bytes Sent By Job

    165274896 Â- ÂRun Bytes Received By Job

**********************************************

(2) Error message on submit node stdout:

[this is actually the TCP reappearance of the worker after the disconnection]

***********************************************

New worker has appeared: slot6@node3

Fortran runtime error: cannot allocate memory