[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] Condor SOAP bug: stopping server when there are pending transactions hangs daemons
- Date: Tue, 2 May 2006 11:39:36 -0500
- From: Matthew Farrellee <matt@xxxxxxxxxxx>
- Subject: Re: [Condor-users] Condor SOAP bug: stopping server when there are pending transactions hangs daemons
Someone with a better understanding of Condor internals, please let
me know if I'm wrong here.
On May 2, 2006, at 9:30 AM, David E. Konerding wrote:
Matthew Farrellee wrote:
On May 1, 2006, at 5:42 PM, David E. Konerding wrote:
I am noticing a very inconvenient bug with Condor SOAP:
If a transaction is begun, and has not yet expired, stopping the
master causes all the daemons to go to a zombie
state and hang around.
This is probably the same problem as the condor_q issue below. All
Condor daemons are single threaded, so if there is a SOAP transaction
active no one can talk to the Schedd. I'm guessing that the Master
just gives up trying to tell it's children to shutdown at some point
and exits. If the children are shutdown serially then a "hanging"
Schedd at the beginning of the child list would account for this.
I'm confused by this answer. There is nothing in a single threaded
application which prevents a server
from maintaining more than one simultaneous transaction (database
servers do this all the time). Nor is there anything
that prevents a server from listening to a port and responding to
multiple requests (nearly) simultaneously.
This is certainly true, and I think I was wrong with what I said
about the port being an issue.
So does this mean that the Condor source base itself has the
of one transaction at a time?
Right now, I believe it does. This is often not a problem, and you
normally would not see this because in local submissions data
transfer is deferred and therefore the operation is very quick. You
should try doing a remote condor_submit (use the -r or -s options)
twice, simultaneously with a large input file. I think you'll find
that the submits are serialized.
What's happening when condor_q is being run at multiple times, or
run when something is being submitted:
is condor_submit using transactions internally, and condor_q blocks
while submits are in progress?
I believe that the condor_q will definitely block. I think the same
thing will happen if you have a large number of job in a queue and
you run two condor_q operations (say start one "condor_q" and then
another "condor_q -constraint FALSE").
Finally, what does this mean for me writing a web service job
and monitor where multiple submitters and monitors will be
accessing the same Condor SOAP server? From my perspective, it means
all my client codes have to be aware of the single-transaction
has to retry operations, and be aggressive about asking for long
transaction times (because I'm doing file transfers and there could be
I don't want to lose an entire job submission and file transfer
transaction just because there was a network dropout), yet
be careful to close down those transactions. If a single client
with a long transaction outstanding, it'll host all the other clients.
It means that you have to deal with a failure case at
BeginTransaction, where the "maximum number of transactions may have
been exceeded." I put that in quotes because until some patches go in
(in the next few weeks) the max number of transactions is set to 1.
This error case is no different than a connection limit on a database
-- a case everyone seems to ignore on the web.
As for an aggressive/long transaction timeout, keep in mind that each
call you make to the Schedd in a transaction will extend your
transaction for the number of seconds you passed to BeginTransaction.
So say you call BeginTransaction(30) and 25 seconds later you call
NewCluster(), once the Schedd receives NewCluster() you have 30
seconds again before your transaction may be aborted, not 5. This
means you can keep a relatively low timeout and still carry out a lot
of work, without fear that a faulty client can hose everyone.
I hope this helps.