[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Random job in DAGman throwing submitERROR



Hi Vikrant,

I might be wrong (the wife assures I often am) but there is a problem that is documented with late materialization and DAGs:

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=6946

The error message I did see in the case of rescheduling was different from your case but if your problem is actually about rescheduling inside a DAG it might be related ...

Best
Christoph


--
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx


Von: "Vikrant Aggarwal" <ervikrant06@xxxxxxxxx>
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
Gesendet: Mittwoch, 15. Mai 2019 09:23:02
Betreff: Re: [HTCondor-users] Random job in DAGman throwing submitERROR

Hello Team,
Any more thoughts on this issue?

Thanks & Regards,
Vikrant Aggarwal


On Sat, May 11, 2019 at 5:11 PM Vikrant Aggarwal <ervikrant06@xxxxxxxxx> wrote:
Thanks for your response. No failing disk on the node. 

There is no schedlog for 04:01:16 when the job submission was failed. 

05/08/19 04:01:15 (pid:1316698) Shadow pid 1055394 for job 147153.3 reports job exit reason 100.
05/08/19 04:01:15 (pid:1316698) Shadow pid 1054140 for job 147147.1 reports job exit reason 100.
05/08/19 04:01:15 (pid:1316698) ERROR fetching job (147154.6) status in check_zombie !
05/08/19 04:01:15 (pid:1316698) Shadow pid 1053429 for job 147147.4 exited with status 100
05/08/19 04:01:15 (pid:1316698) Shadow pid 1053482 for job 147151.3 exited with status 100
05/08/19 04:01:15 (pid:1316698) Shadow pid 1056671 for job 147155.1 exited with status 100
05/08/19 04:01:15 (pid:1316698) Shadow pid 1056674 for job 147155.4 exited with status 100
05/08/19 04:01:15 (pid:1316698) Number of Active Workers 0
05/08/19 04:01:15 (pid:1316698) Number of Active Workers 0
05/08/19 04:01:17 (pid:1316698) Activity on stashed negotiator socket: <IPaddress:32651>
05/08/19 04:01:17 (pid:1316698) Using negotiation protocol: NEGOTIATE

This is the main logic used in dagman submit file. 

remove_kill_sig = SIGUSR1
+OtherJobRemoveRequirements     = "DAGManJobId =?= $(cluster)"
# Note: default on_exit_remove _expression_:
# ( ExitSignal =?= 11 || (ExitCode =!= UNDEFINED && ExitCode >=0 && ExitCode <= 2))
# attempts to ensure that DAGMan is automatically
# requeued by the schedd if it exits abnormally or
# is killed (e.g., during a reboot).
on_exit_remove  = (ExitSignal =?= 11 || (ExitCode =!= UNDEFINED && ExitCode >=0 && ExitCode <= 2))
copy_to_spool   = False

Jobs "147157.4" and "147154.6" are part of same dagman output. 

# grep -ir '147154.6' condor_20190507.20190508.033002.618997.dag.dagman.out
05/08/19 03:59:28 Reassigning the id of job test_21090507_ from (147154.5.0) to (147154.6.0)
05/08/19 03:59:28 Event: ULOG_SUBMIT for HTCondor Node test_21090507_ (147154.6.0) {05/08/19 03:59:27}
05/08/19 03:59:28 Reassigning the id of job test_21090507_ from (147154.6.0) to (147154.7.0)
05/08/19 03:59:45 Event: ULOG_EXECUTE for HTCondor Node test_21090507_ (147154.6.0) {05/08/19 03:59:28}
05/08/19 04:01:16 Event: ULOG_JOB_TERMINATED for HTCondor Node test_21090507_ (147154.6.0) {05/08/19 04:00:39}
05/08/19 04:01:16 Node test_21090507_ job proc (147154.6.0) completed successfully.

# grep -ir '147157.4' condor_20190507.20190508.033002.618997.dag.dagman.out                           
05/08/19 04:01:16 From submit: ERROR: Failed submission for job 147157.4 - aborting entire submit
05/08/19 04:01:16 Read so far: Submitting job(s)....ERROR: Failed submission for job 147157.4 - aborting entire submitERROR: Failed to queue job.


Thanks & Regards,
Vikrant Aggarwal


On Thu, May 9, 2019 at 11:04 PM John M Knoeller <johnkn@xxxxxxxxxxx> wrote:

Itâs still not the right time period of the SchedLog.    and the error about Zombie checking is for different job, itâs not even the same cluster.

 

You might check to see if you have a failing disk, that *might* explain both of these problems.   I canât think of anything else that could.

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Vikrant Aggarwal
Sent: Thursday, May 9, 2019 10:40 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Random job in DAGman throwing submitERROR

 

Sorry for typo. Snippet was from schedlogs only not shadow.

On Thu, 9 May, 2019, 21:07 John M Knoeller, <johnkn@xxxxxxxxxxx> wrote:

The ShadowLog doesnât isnât where to look for this error.  when submit fails, there will never be a shadow for that job.

you should look in the SchedLog at time 05/08/19 04:01:16 to see if it has a reason why the submit failed.

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Vikrant Aggarwal
Sent: Thursday, May 9, 2019 3:15 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Random job in DAGman throwing submitERROR

 

Hello Team,

 

We are facing weird issue with DAGman consists of 54 jobs. One of the job in DAG is randomly throwing an error at no particular frequency.  I am trying to debug the reason for same. 

 

05/08/19 04:01:16 From submit: Submitting job(s)....
05/08/19 04:01:16 From submit: ERROR: Failed submission for job 147157.4 - aborting entire submit
05/08/19 04:01:16 From submit:
05/08/19 04:01:16 From submit: ERROR: Failed to queue job.
05/08/19 04:01:16 failed while reading from pipe.
05/08/19 04:01:16 Read so far: Submitting job(s)....ERROR: Failed submission for job 147157.4 - aborting entire submitERROR: Failed to queue job.
05/08/19 04:01:16 ERROR: submit attempt failed

 

Shadow logs are not showing any indication for this job but it does show the "status in check_zombie" message for another job of same Dag. Most of the time I noticed this zombie message appearing in sched logs during the time of issue but it's not everytime. 

 

05/08/19 04:01:15 (pid:1316698) Shadow pid 1054140 for job 147147.1 reports job exit reason 100.
05/08/19 04:01:15 (pid:1316698) ERROR fetching job (147154.6) status in check_zombie !
05/08/19 04:01:15 (pid:1316698) Shadow pid 1053429 for job 147147.4 exited with status 100


condor version details 

 

$CondorVersion: 8.5.8 Dec 13 2016 BuildID: 390781 $

$CondorPlatform: x86_64_RedHat6 $

 

Anyone else saw this issue? 

 

Thanks & Regards,

Vikrant Aggarwal

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/