Mailing List ArchivesPublic Access |
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
|
Mailing List ArchivesPublic Access |
|
|
Greetings,
I am running Kubuntu15.10. I have noticed that when I log out, all of my jobs stop running and migrate back into the queue. When I log back in schedd is not running and I have to manually restart condor with systemctl. I'm relatively new at debugging condor, so let me know what log files would help me diagnose this problem. Below is an exceprt from SchedLog. NOTE: I logged out at 16:48.
01/07/16 16:48:39 (pid:3782) Got SIGQUIT. Performing fast shutdown. 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4081] for job 11.21 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6633] for job 11.63 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6637] for job 11.64 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4349] for job 11.51 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4350] for job 11.52 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6394] for job 57.0 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4354] for job 11.53 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4358] for job 11.54 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3848] for job 11.73 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6399] for job 60.0 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3849] for job 11.19 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4362] for job 11.55 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 9496] for job 11.103 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4419] for job 11.46 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3911] for job 11.1 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3912] for job 11.2 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3913] for job 11.0 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3917] for job 11.3 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3921] for job 11.4 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8768] for job 11.65 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4178] for job 11.22 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8769] for job 11.66 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4179] for job 11.23 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4180] for job 11.26 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3925] for job 11.5 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4181] for job 11.25 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8773] for job 11.68 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3929] for job 11.7 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8777] for job 11.69 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3933] for job 11.6 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8781] for job 11.70 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3936] for job 11.8 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4191] for job 11.24 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4192] for job 11.27 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4193] for job 11.28 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8785] for job 11.71 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3941] for job 11.10 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8789] for job 11.72 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3944] for job 11.9 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3946] for job 11.11 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4202] for job 11.29 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8793] for job 11.74 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4203] for job 11.30 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8797] for job 11.75 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4207] for job 11.31 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6503] for job 11.45 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6504] for job 11.56 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8801] for job 11.76 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3956] for job 11.12 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8805] for job 11.77 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4215] for job 11.32 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 3961] for job 11.13 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8809] for job 11.78 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4219] for job 11.33 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4223] for job 11.34 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8814] for job 11.79 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4227] for job 11.35 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8818] for job 11.80 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4231] for job 11.36 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8822] for job 11.81 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4235] for job 11.37 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8826] for job 11.82 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8830] for job 11.83 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4241] for job 11.38 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8834] for job 11.84 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4245] for job 11.39 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8838] for job 11.85 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4250] for job 11.40 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8842] for job 11.87 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4254] for job 11.41 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8846] for job 11.88 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4258] for job 11.42 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8850] for job 11.90 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8854] for job 11.91 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4264] for job 11.43 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8859] for job 11.92 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4271] for job 11.44 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4272] for job 11.47 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8863] for job 11.96 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4276] for job 11.48 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8867] for job 11.97 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 20089] for job 11.122 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4280] for job 11.49 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8871] for job 11.98 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4284] for job 11.50 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8875] for job 11.99 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8879] for job 11.100 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8883] for job 11.101 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 8887] for job 11.102 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4060] for job 11.14 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4061] for job 11.15 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4062] for job 11.16 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6613] for job 11.57 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6614] for job 11.58 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 12738] for job 11.104 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6618] for job 11.59 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4069] for job 11.18 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4070] for job 11.17 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6622] for job 11.60 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6625] for job 11.61 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 4076] for job 11.20 01/07/16 16:48:39 (pid:3782) Sent signal 9 to shadow [pid 6629] for job 11.62 01/07/16 16:48:39 (pid:3782) Cron: Killing all jobs 01/07/16 16:48:39 (pid:3782) All shadows have been killed, exiting. 01/07/16 16:48:39 (pid:3782) **** condor_schedd (condor_SCHEDD) pid 3782 EXITING WITH STATUS 0 01/08/16 07:47:16 (pid:18292) Setting maximum file descriptors to 4096.
The next morning when I get back to work, this is the startup log for Sched after I manually restarted condor via systemctl.
01/08/16 07:47:16 (pid:18292) ****************************************************** 01/08/16 07:47:16 (pid:18292) ** condor_schedd (CONDOR_SCHEDD) STARTING UP 01/08/16 07:47:16 (pid:18292) ** /usr/sbin/condor_schedd 01/08/16 07:47:16 (pid:18292) ** SubsystemInfo: name=SCHEDD type=SCHEDD(5) class=DAEMON(1) 01/08/16 07:47:16 (pid:18292) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON 01/08/16 07:47:16 (pid:18292) ** $CondorVersion: 8.2.9 Aug 17 2015 BuildID: Debian-8.2.9~dfsg.1-1 Debian-8.2.9~dfsg.1-1 $ 01/08/16 07:47:16 (pid:18292) ** $CondorPlatform: X86_64-Ubuntu_ $ 01/08/16 07:47:16 (pid:18292) ** PID = 18292 01/08/16 07:47:16 (pid:18292) ** Log last touched 1/7 16:48:39 01/08/16 07:47:16 (pid:18292) ******************************************************
Any help you guys can provide is much appreciated.
Michael McInerny Murphy Engineer & Physicist IERUS Technologies, Inc. 2904 Westcorp Blvd, Ste. 210 Office: (256) 319-2026 Ext 107 |