[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Condor-users] Unwanted job eviction



Hi,
in a dedicated cluster running Condor 6.6.8 many jobs get evicted from a node and restarted shortly after on another node.
The jobs cannot be checkpointed and they are either restarting from scratch or sometime failing, if data recorded in a NFS mounted job directory by the previous attempt is inconsistent.
Is it there any way to understand why?
Is it possible to disable this behavior?
Below is an example of condor job log of one of the evicted jobs.


Thanks,
Marco


000 (7707.000.000) 03/23 01:58:15 Job submitted from host: <10.255.255.254:32806>
...
001 (7707.000.000) 03/23 01:58:18 Job executing on host: <10.255.255.216:32811>
...
006 (7707.000.000) 03/23 01:58:26 Image size of job updated: 25884
...
006 (7707.000.000) 03/23 02:18:26 Image size of job updated: 541852
...
...............
...
006 (7707.000.000) 03/23 09:58:26 Image size of job updated: 562936
...
004 (7707.000.000) 03/23 10:39:50 Job was evicted.
(0) Job was not checkpointed.
Usr 0 17:02:20, Sys 0 00:00:31 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
...
001 (7707.000.000) 03/23 10:43:55 Job executing on host: <10.255.255.230:40512>
...
006 (7707.000.000) 03/23 11:04:04 Image size of job updated: 541776
...