[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Condor-users] My dag is frozen




Hi Steffen and condor experts,

for the last 4 days my dagman in morgane seems frozen and doesn't trigger more jobs. The dag is not yet finished, as you can see if you execute

lucia@morgane:~/month4_latest/857232370-859651570$ /home/lucia/opt/s5_2yr_lowcbc_20080603/lalapps//bin/lalapps_ihope_status --dag-file=ihope.dag

(playground is done up to cat4, injection jobs only up to cat1)

however nothing happens

lucia@morgane:~/month4_latest/857232370-859651570$ condor_q lucia
-- Submitter: morgane.merlin2.aei.mpg.de : <10.100.200.91:39685> : morgane.merlin2.aei.mpg.de
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
70996.0 lucia 7/2 15:20 15+19:13:20 R 0 7.3 condor_dagman -f - 167274.0 lucia 7/12 23:16 5+15:29:59 R 0 7.3 condor_dagman -f -

and also, _no_ rescue dag is created. If I look at one of the dag.dagman.out corresponding to one of the subdags that are not yet finished (for instance, cat2 dags in nsbhinj):

--> in nbshinj/inspiral_hipe_nsbhinj_cat2_veto.NSBHINJ_CAT_2_VETO.dag.dagman.out
(tail of the file ... for 4 days now)
7/18 14:05:26 Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, status STATUS_SUBMITTED
7/18 14:15:26 319886 seconds since last log event
7/18 14:15:26 Pending DAG nodes:
7/18 14:15:26 Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, status STATUS_SUBMITTED
7/18 14:25:26 320486 seconds since last log event
7/18 14:25:26 Pending DAG nodes:
7/18 14:25:26 Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, status STATUS_SUBMITTED
7/18 14:35:26 321086 seconds since last log event
7/18 14:35:26 Pending DAG nodes:
7/18 14:35:26 Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, status STATUS_SUBMITTED

Anybody has a clue what might be happening?
Thanks,
Lucia

--
--------------------------------------------
Lucia Santamaria
Max-Planck-Institut fuer Gravitationsphysik
Albert-Einstein-Institut
Am Muehlenberg 1, 17746 Golm, Germany
Office: +49(0)331-567-7181
---------------------------------------------