Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] 7.8.2 / running out of ports for UDP

Try setting NETWORK_INTERFACE = <IP address>, that might fix it. Note that interface names won't work, at least they didn't when I ran into this. 

https://lists.cs.wisc.edu/archive/condor-users/2012-June/msg00043.shtml

On 09/06/2012 06:31 PM, Shrum, Donald C wrote:

Yes.  In fact I am running with no dns.

-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Rob de Graaf
Sent: Thursday, September 06, 2012 12:26 PM
To: Condor-Users Mail List
Subject: Re: [Condor-users] 7.8.2 / running out of ports for UDP

Just a guess, but are you running with NO_DNS?

https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2691,3

On 09/06/2012 06:09 PM, Dan Bradley wrote:


Is there anything interesting in MasterLog?

It may be helpful to strace condor_master.

strace -p <insert-pid-of-master> -o master.strace

You can kill that after it has run for long enough to observe lots of
sockets being opened.

What is using port 1980?  Have you configured your collector to use
that port?

--Dan

On 9/6/12 11:02 AM, Shrum, Donald C wrote:


Hi Dan,

The problem persists so flocking was not related.  The condor master
is back up to 28,000+ open UDP ports.

Right now I'm just doing a periodic restart

[root@condor ~]# condor_status -master -format "%d\n"
MonitorSelfRegisteredSocketCount

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

[root@condor ~]# service condor restart

Shutting down Condor (fast-shutdown mode)...  done.

Starting up Condor...    done.

[root@condor ~]# condor_status -master -format "%d\n"
MonitorSelfRegisteredSocketCount

1

Thanks for the help -

Don

FSU HPC

*From:*condor-users-bounces@xxxxxxxxxxx
[mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Dan Bradley
*Sent:* Thursday, September 06, 2012 10:53 AM
*To:* condor-users@xxxxxxxxxxx
*Subject:* Re: [Condor-users] 7.8.2 / running out of ports for UDP

Donald,

If you observe this problem again, see what the daemon is reporting
in its ClassAd:

condor_status -master -format "%d\n" MonitorSelfRegisteredSocketCount
<insert-hostname-here>

What's using port 1980?  The collector?

--Dan

On 9/6/12 9:28 AM, Shrum, Donald C wrote:

     As always, thanks Ian.

     We had flocking set up with another University and using the 'this
     was one of the last things I touched' trouble shooting method I
     just disabled flocking and Condor Connection Brokering
(CCB_ADDRESS)

     That may have resolved the problem... we'll see.

     --Don

     FSU HPC

     *From:*condor-users-bounces@xxxxxxxxxxx
     <mailto:condor-users-bounces@xxxxxxxxxxx>
     [mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Ian Chesal
     *Sent:* Thursday, September 06, 2012 9:34 AM
     *To:* Condor-Users Mail List
     *Subject:* Re: [Condor-users] 7.8.2 / running out of ports for
UDP

     Donald,

     You could switch to TCP for collector updates:

     UPDATE_COLLECTOR_WITH_TCP = True

     See:

http://research.cs.wisc.edu/condor/manual/v7.6/3_3Configuration.html#
16701

     Or even better: switch to using the shared port daemon. This
     should help reduce the number of connections needed on any one
     machine. See:

http://research.cs.wisc.edu/condor/manual/v7.6/3_7Networking_includes
.html#32152

     Regards,

     - Ian

     --

     Ian Chesal

     Cycle Computing, LLC

     Leader in Open Compute Solutions for Clouds, Servers, and
Desktops

     Enterprise Condor Support and Management Tools

     888.292.5320

     http://www.cyclecomputing.com <http://www.cyclecomputing.com/>

     http://www.cyclecloud.com <http://www.cyclecloud.com/>

     http://twitter.com/cyclecomputing

     On Thursday, 6 September, 2012 at 9:28 AM, Shrum, Donald C wrote:

         Looks like the collector -

         udp 0 0 10.178.6.5:41796 10.178.6.5:1980 ESTABLISHED
         580/condor_collecto

         udp 0 0 10.178.6.5:43588 10.178.6.5:1980 ESTABLISHED
         580/condor_collecto

         udp 0 0 10.178.6.5:48964 10.178.6.5:1980 ESTABLISHED
         580/condor_collecto

         udp 0 0 10.178.6.5:40004 10.178.6.5:1980 ESTABLISHED
         580/condor_collecto

         udp 0 0 10.178.6.5:47684 10.178.6.5:1980 ESTABLISHED
         580/condor_collecto

         This was on the central manager. Next time I see it happen on
         a processing node I'll check there as well.

         -----Original Message-----

         From: condor-users-bounces@xxxxxxxxxxx
         <mailto:condor-users-bounces@xxxxxxxxxxx>
         [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Brian
         Candler

         Sent: Thursday, September 06, 2012 8:36 AM

         To: Condor-Users Mail List

         Subject: Re: [Condor-users] 7.8.2 / running out of ports for
UDP

         On Thu, Sep 06, 2012 at 12:27:46PM +0000, Shrum, Donald C wrote:

             I'm running redhat 6.3 with condor 7.8.2

             On a number of my servers, both processing and on the
             central manager;

             I find condor holding open a massive number of UDP ports.
             So many that

             it blocks any new connections and DNS lookups fail.

             Is this happening for anyone else?

         Can you say which particular condor process is holding open
         the ports?

         netstat -naup

         (as root) should show you the process name and pid for each
         socket.

         _______________________________________________

         Condor-users mailing list

         To unsubscribe, send a message to
         condor-users-request@xxxxxxxxxxx
         <mailto:condor-users-request@xxxxxxxxxxx> with a

         subject: Unsubscribe

         You can also unsubscribe by visiting

         https://lists.cs.wisc.edu/mailman/listinfo/condor-users

         The archives can be found at:

         https://lists.cs.wisc.edu/archive/condor-users/

         _______________________________________________

         Condor-users mailing list

         To unsubscribe, send a message to
         condor-users-request@xxxxxxxxxxx
         <mailto:condor-users-request@xxxxxxxxxxx> with a

         subject: Unsubscribe

         You can also unsubscribe by visiting

         https://lists.cs.wisc.edu/mailman/listinfo/condor-users

         The archives can be found at:

         https://lists.cs.wisc.edu/archive/condor-users/




     _______________________________________________

     Condor-users mailing list

     To unsubscribe, send a message tocondor-users-request@xxxxxxxxxxx
<mailto:condor-users-request@xxxxxxxxxxx>  with a

     subject: Unsubscribe

     You can also unsubscribe by visiting

     https://lists.cs.wisc.edu/mailman/listinfo/condor-users



     The archives can be found at:

     https://lists.cs.wisc.edu/archive/condor-users/



_______________________________________________
Condor-users mailing list
To unsubscribe, send a message tocondor-users-request@xxxxxxxxxxx
with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/




_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/


_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/


_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/