Re: [Condor-users] Condor job submission delayed

[Alain Roy <roy@xxxxxxxxxxx>]

Marc Saric wrote: (I'm collating multiple emails.)

> Submission of jobs works in principle (tested it with the
> hello-world-examples from http://www.liv.ac.uk/e-science/condor/hello.html
> but sometimes I observe a strange behaviour in that certain jobs need a
> very long time until they are beeing executed.

> Around 30 min.

That is definitely too long, unless you have thousands of jobs and 
computers, and even then it's too long.

> 045.000:  Run analysis summary.  Of 12 machines,
> ~      1 are rejected by your job's requirements
> ~      6 reject your job because of their own requirements
> ~      0 match, but are serving users with a better priority in the pool
> ~      4 match, match, but reject the job for unknown reasons

That message is unfortunately pretty undescriptive. Yes, we need to improve 
it. There are a few common things that this may indicate:

1) The negotiator is currently matching jobs to machines, and this is a 
transitory state. I don't think this is the case for you.

2) You have a worse priority than another user on the system. Again, I 
don't think this is the case.

3) Ummm... Something else. A lot of things can go wrong, and it's really 
hard to properly figure them out from condor_q.

Given that your jobs eventually match and you have a small, mostly idle 
pool, this message seems odd.

> | 300 second delays can occur if the new job started while condor was
> | within a 20 seconds frame of the negotiation cycle. you can start the
> | job by using condor_reschedule.
> | you can reduce the time by lowering the NEGOTIATOR_INTERVAL value, but
> | the 20 seconds timeframe is fixed, so for a 60 second interval you have
> | a 33% chance that your job must wait up to a minute.
>
> That's clear to me, I did not expect the queued jobs to be executed
> within a few seconds, but (recaling my first mail) it was unclear to me,
> why sometimes (not allways) jobs don't get executed for 10-30 minutes
> while "condor_status" lists a lot of machines (all of them Windows in my
> case) as Unclaimed/Idle during that period.

The negotiator will do a matchmaking cycle every five minutes, unless you 
submit a job with condor_submit: then it will start a new cycle, unless the 
last cycle began less than twenty seconds previously. If you are submitting 
individual test jobs, then jobs should be matched quickly. Not instantly: 
it always takes them a while to start up, but certainly much quicker than 
30 minutes. I think that NEGOTIATOR_INTERVAL won't help you here.

Here are a couple of plausible things we should look at:

1) Sometimes a job does match quickly and get submitted to a computer, but 
as soon as it starts up, an error is encountered and the job dies. Unless 
you do condor_q at exactly the right moment, it seems to always be idle, 
but it was in the running state for a brief moment.

You can see if this is the case: look in the user log for the job 
(specified with the "log = X" in your submit file) and see if it says the 
job was started up multiple times. If it was, then we need to figure out 
why it died when starting up. If this is the case, please share your user 
log. If not, try #2:

2) Try this: submit a new job and see what time you submitted it. Let it 
sit idle for a few minutes, then do "condor_reschedule" to force another 
matchmaking cycle. Now go to the computer that is running the negotiator, 
look in the NegotiatorLog and the MatchLog, beginning at the time that you 
submitted the job. Share this portion of the logs with me and/or the 
mailing list (I don't want the entire log--just from the time you started 
the test). This may indicate why there is a problem.

3) I've seen an intermittent problem when people submit a single job and it 
doesn't get matched, but if they submit multiple jobs it does work. This 
happened in rare circumstance, and I'm pretty sure it was fixed. That said, 
try submitting two or three jobs, and see if your jobs run.

I suspect that #1 or #2 will help us narrow down the problem. If not, we 
can figure out something else that will help.

-alain