[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Condor is still writing temp data to /var/opt/condor



Thanks again for bearing with me. So "condor_status -master -af Name MyAddress" also didn't work (Same error about -af).

I've never logged into my compute nodes. Someone mentioned to me that I could do it from head node, but I don't know the command for it. Again, I'm running RHEL6 Server on my head node. 

Nate Mobley
Millennium Engineering & Integration Company
ISSO/Systems Administrator
Desk: (256) 489-7847
Cell (Voice Only): (256) 655-5570
MEI Help Desk:  (703) 413-7771
nmobley@xxxxxxxxxxxxxx
www.meicompany.com


-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of htcondor-users-request@xxxxxxxxxxx
Sent: Tuesday, January 09, 2018 4:15 PM
To: htcondor-users@xxxxxxxxxxx
Subject: HTCondor-users Digest, Vol 50, Issue 26

Send HTCondor-users mailing list submissions to
	htcondor-users@xxxxxxxxxxx

To subscribe or unsubscribe via the World Wide Web, visit
	https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
or, via email, send a message with subject or body 'help' to
	htcondor-users-request@xxxxxxxxxxx

You can reach the person managing the list at
	htcondor-users-owner@xxxxxxxxxxx

When replying, please edit your Subject line so it is more specific than "Re: Contents of HTCondor-users digest..."


Today's Topics:

   1. Re: Condor is still writing temp data to /var/opt/condor
      (John M Knoeller)
   2. Re: Error with global Queue (Brandon Graves)


----------------------------------------------------------------------

Message: 1
Date: Tue, 9 Jan 2018 22:05:26 +0000
From: John M Knoeller <johnkn@xxxxxxxxxxx>
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>,	Todd
	Tannenbaum	<tannenba@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Condor is still writing temp data to
	/var/opt/condor
Message-ID:
	<CY1PR0601MB2021112454C7BFFC9D30C27D96100@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
	
Content-Type: text/plain; charset="utf-8"

-af is short for -autoformat, and it's definitely a valid option for condor_status.  the problem is probably that complext regexps() expression.  try just using

condor_status -master -af Name MyAddress


-tj

-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Mobley, Nate (Millennium)
Sent: Tuesday, January 9, 2018 3:51 PM
To: Todd Tannenbaum <tannenba@xxxxxxxxxxx>; HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Condor is still writing temp data to /var/opt/condor

Thanks, not a problem getting my node names to display. However, when I try to connect via SSH with Putty, it tells me "Unable to open connection to "hostname.local" Host not found."

When I ran "condor_status -master -af Name 'regexps("<([^:]*)",MyAddress,"\1")'" I received the error "Unknown option -af" and when I eliminated the "-af," the command ran but didn't return anything...

I appreciate your time! 

Nate Mobley
Millennium Engineering & Integration Company ISSO/Systems Administrator
Desk: 256-489-7847
Cell (Voice Only): 256-655-5570
MEI Help Desk:? 703-413-7771
nmobley@xxxxxxxxxxxxxx
www.meicompany.com


-----Original Message-----
From: Todd Tannenbaum [mailto:tannenba@xxxxxxxxxxx] 
Sent: Tuesday, January 09, 2018 2:02 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>; Mobley, Nate (Millennium) <nmobley@xxxxxxxxxxxxxx>
Subject: Re: [HTCondor-users] Condor is still writing temp data to /var/opt/condor

On 1/9/2018 12:36 PM, Mobley, Nate (Millennium) wrote:
> This is great news, now I think I know exactly what's causing the data to still go to "/var/opt/condor". My challenge now is how to change the config file on each compute node...I've never logged into them before. I'm having trouble determining what their IP addresses are, or else I think I could SSH into them with putty and modify the config files. I assume the " condor_config_val -v -startd -name compute-0-0.local EXECUTE" will show me the IP addresses, so I'll start there. Thank you all for your time.
> 

Run "condor_status -master" to see all the hosts (nodes) in your pool.   Note this will display hostnames, like:

   Name                 Version        Cpus   Memory      Uptime    

   node1.wisc.edu       8.6.9.427508      8    15.5 GB   20+02:31:06
   node2.wisc.edu       8.6.9.427508      8     7.7 GB   20+02:31:41
   node3.wisc.edu       8.6.9.427508      8    15.5 GB   20+02:31:58


If you really need IP addresses instead of hostnames, you could do something like:

   condor_status -master -af Name 'regexps("<([^:]*)",MyAddress,"\1")'

which will give output like:

   node1.wisc.edu 10.0.5.22
   node2.wisc.edu 10.0.5.23
   node3.wisc.edu 10.0.5.24
   ..


regards
Todd




> Nate Mobley
> Millennium Engineering & Integration Company ISSO/Systems 
> Administrator
> Desk: 256-489-7847
> Cell (Voice Only): 256-655-5570
> MEI Help Desk:? 703-413-7771
> nmobley@xxxxxxxxxxxxxx
> www.meicompany.com
> 
> 
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On 
> Behalf Of htcondor-users-request@xxxxxxxxxxx
> Sent: Tuesday, January 09, 2018 12:00 PM
> To: htcondor-users@xxxxxxxxxxx
> Subject: HTCondor-users Digest, Vol 50, Issue 24
> 
> Send HTCondor-users mailing list submissions to
> 	htcondor-users@xxxxxxxxxxx
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> or, via email, send a message with subject or body 'help' to
> 	htcondor-users-request@xxxxxxxxxxx
> 
> You can reach the person managing the list at
> 	htcondor-users-owner@xxxxxxxxxxx
> 
> When replying, please edit your Subject line so it is more specific than "Re: Contents of HTCondor-users digest..."
> 
> 
> Today's Topics:
> 
>     1. Re: Condor is still writing temp data to /var/opt/condor even
>        though I specified otherwise (John M Knoeller)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Tue, 9 Jan 2018 17:05:28 +0000
> From: John M Knoeller <johnkn@xxxxxxxxxxx>
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>,	Todd
> 	Tannenbaum	<tannenba@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] Condor is still writing temp data to
> 	/var/opt/condor even though I specified otherwise
> Message-ID:
> 	
> <CY1PR0601MB2021D37279EF641838EAB62A96100@xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> d.outlook.com>
> 	
> Content-Type: text/plain; charset="iso-8859-1"
> 
> Yes, there is a separate config for each node.  Also, daemons will only see changes to the config when you condor_reconfig (or restart) them.
> 
> The - before the name of your compute node is messing things up. you want this command.
> 
> condor_config_val -v -startd -name compute-0-0.local EXECUTE
> 
> not this
> 
> condor_config_val -v -startd -compute-0-0.local EXECUTE
> 
> The -startd and -name <name> arguments tell condor_config_val to ask the condor_startd daemon what its current config is.   This will be what the config files on that node contained at the time the condor_startd was last started or reconfiged - which might be older than what the config files say now.
> 
> -tj
> 
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On 
> Behalf Of Mobley, Nate (Millennium)
> Sent: Tuesday, January 9, 2018 10:20 AM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>; Todd 
> Tannenbaum <tannenba@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] Condor is still writing temp data to 
> /var/opt/condor even though I specified otherwise
> 
> Thanks again Todd. So when I ran " condor_config_val -v execute" it did show me the file I was expecting. But I got to thinking after that, is there a separate config file for each node?
> 
> I'm unfamiliar with what the config knob is, could you explain that a 
> little more? I did run " condor_config_val -v -startd 
> -compute-0-0.local" (where compute-0-0.local is the name of one of my 
> compute nodes) and the error I received was "Can't connect to startd 
> on head1.domainname.com <192.***.***.*: *****>" (domainname instead of 
> my actual domain name)
> 
> 
> Nate Mobley
> Millennium Engineering & Integration Company ISSO/Systems 
> Administrator
> Desk: (256) 489-7847
> Cell (Voice Only): (256) 655-5570
> MEI Help Desk:? (703) 413-7771
> nmobley@xxxxxxxxxxxxxx
> www.meicompany.com
> 
> 
> -----Original Message-----
> From: Todd Tannenbaum [mailto:tannenba@xxxxxxxxxxx]
> Sent: Monday, January 08, 2018 4:57 PM
> To: Mobley, Nate (Millennium) <nmobley@xxxxxxxxxxxxxx>
> Cc: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] Condor is still writing temp data to 
> Condor is still writing temp data to /var/opt/condor even though I 
> specified otherwise
> 
> On 1/8/2018 4:55 PM, Todd Tannenbaum wrote:
>> On 1/8/2018 4:25 PM, Mobley, Nate (Millennium) wrote:
>>> Thanks. They were not overwritten, Ive already triple checked that.
>>> Can I run "condor_config_val -v execute" from my head node? That is 
>>> the only one I am able to log into (I don't know the IP addresses of 
>>> the compute nodes).
>>>
>>
>> If by "head node" you mean the HTCondor Central Manager, then yes, you can use condor_config_val to view configuration settings for any node in your compute pool (unless your installation changed the default authorization rules).
>>
>> You will want to use
>>     condor_config_val -v -startd -name <name_of_execute_node> 
>> <name_of_config knob>
>>
>> So to see your execute nodes, enter "condor_status", and then use a name you see there with condor_config_val.
>>
>> So for example, the sample session below tells me my execute nodes 
>> apparently have EXECUTE undefined, which means they will use the 
>> default value of $(LOCAL_DIR)/execute, and LOCAL_DIR is set to /var in condor_config.global on line 26.
>>
>>
>>     $ condor_status -wide | head
>>     Name                               OpSys      Arch   State     Activity     LoadAv Mem      ActvtyTime
>>
>>     slot1@xxxxxxxxxxxxxxx              LINUX      X86_64 Unclaimed Idle          0.000   43883  0+05:13:41
>>     slot1@xxxxxxxxxxxxxxx              LINUX      X86_64 Unclaimed Idle          0.000   43883  0+05:13:41
>>     slot1@xxxxxxxxxxxxxxx              LINUX      X86_64 Unclaimed Idle          0.000   43883  0+05:13:41
>>     ...
>>
>>     $ condor_config_val -v -startd -name slot1@xxxxxxxxxxxxxxx EXECUTE
>>     EXECUTE =
>>      # at: /etc/condor/condor_config.global, line 44
>>      # raw: EXECUTE =
>>      # default: $(LOCAL_DIR)/execute
>>
>>     $ condor_config_val -v -startd -name slot1@xxxxxxxxxxxxxxx 
>> EXECUTE
> 
> Meant this line to be
> 
> condor_config_val -v -startd -name slot1@xxxxxxxxxxxxxxx LOCAL_DIR
> 
> regards,
> Todd
> 
> 
> 
>>     LOCAL_DIR = /var
>>      # at: /etc/condor/condor_config.global, line 26
>>      # raw: LOCAL_DIR = /var
>>
>>
>> Hope the above helps
>> Todd
>>
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
>> with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx 
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> HTCondor-users mailing list
> HTCondor-users@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> ------------------------------
> 
> End of HTCondor-users Digest, Vol 50, Issue 24
> **********************************************
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx 
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/



------------------------------

Message: 2
Date: Tue, 09 Jan 2018 14:15:08 -0800
From: Brandon Graves <bmgraves@xxxxxx>
To: htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Error with global Queue
Message-ID: <1ec36c41-717e-a978-ae0f-83a7fcb44a83@xxxxxx>
Content-Type: text/plain; charset="utf-8"

Just for Records sake problem resolved, I added:

*CONDOR_Q_ONLY_MY_JOBS = FALSE*

*
*to my config files, and it resolved the issue.


Thanks for the help on the other errors!

-Brandon


On 1/8/18 3:09 PM, Brandon Graves wrote:
> Alright, I now Have
>
> Central Manager = MASTER, COLLECTOR, NEGOTIATOR
> Submit Nodes = MASTER, SCHEDD
> Execute nodes = MASTER, SCHEDD, STARTD
>
> That has fixed the error in the logs, but "condor_q -global" still presents:
>
>> Failed to fetch ads from:
> <xxx.xxx.xxx.xxx:9618?addrs=xxx.xxx.xxx.xxx-9618&noUDP&sock=3573919_bf24_3>
> : host.my.domain.com
>> AUTHENTICATE:1003:Failed to authenticate with any method
>> AUTHENTICATE:1004:Failed to authenticate using GSI
>> GSI:5003:Failed to authenticate.? Globus is reporting error
> (851968:50).? There is probably a problem with your credentials.? (Did
> you run grid-proxy-init?)
>> AUTHENTICATE:1004:Failed to authenticate using KERBEROS
>> AUTHENTICATE:1004:Failed to authenticate using FS
> Thank's again for your help, if you have any more idea's I'd be very
> appreciative
>
>
> --Brandon
>
> On 1/8/18 1:16 PM, John M Knoeller wrote:
>> This message. 
>>
>>> 01/08/18 09:59:56 DaemonCore: Can't receive command request from
>> xxx.xxx.xxx.105 (perhaps a timeout?)
>>
>> Is generally not a problem.  You will see it in working  pools once per negotiation cycle, it happens because the negotiator hangs up after updating accounting ads, but the collector assumes that any socket opened for updates will never be closed, so it looks for a second command after the first and we get a warning when It doesn't find one.
>>
>> This message. 
>>> 01/08/18 10:03:14 PERMISSION DENIED to condor_pool@xxxxxxxxxxxxx from
>> host xxx.xxx.xxx.60 for command 10 (QUERY_STARTD_PVT_ADS), access level
>> NEGOTIATOR: reason: cached result for NEGOTIATOR; see first case for the
>> full reason
>>
>> Is a problem.  It indicates that host xxx.xxx.xxx.60 is running a NEGOTIATOR, and that negotiator is unable to negotiate because
>> the COLLECTOR will not send it the information it needs to do so.   if xxx.xxx.xxx.60 is not your central manager, then you probably just
>> need to remove NEGOATIATOR from the DAEMON_LIST in the configuration of host xxx.xxx.xxx.60
>>
>> try running
>>
>> condor_config_val -verbose DAEMON_LIST
>>
>> on each of your submit nodes.  the result should not have either COLLECTOR or NEGOTIATOR.  
>>
>> -tj
>>
>> -----Original Message-----
>> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Brandon Graves
>> Sent: Monday, January 8, 2018 12:18 PM
>> To: htcondor-users@xxxxxxxxxxx
>> Subject: Re: [HTCondor-users] Error with global Queue
>>
>> Both commands yield valid results without errors.
>>
>> in the Central Managers CollectorLog I have:
>>
>>> 01/08/18 10:03:14 PERMISSION DENIED to condor_pool@xxxxxxxxxxxxx from
>> host xxx.xxx.xxx.60 for command 10 (QUERY_STARTD_PVT_ADS), access level
>> NEGOTIATOR: reason: cached result for NEGOTIATOR; see first case for the
>> full reason
>>> 01/08/18 10:03:14 DC_AUTHENTICATE: Command not authorized, done!
>>> 01/08/18 10:03:20 Got QUERY_STARTD_ADS
>>> 01/08/18 10:03:20 Number of Active Workers 0
>>> 01/08/18 10:03:20 Got QUERY_STARTD_ADS
>>> 01/08/18 10:03:20 Number of Active Workers 0
>>> 01/08/18 10:03:26 Got QUERY_STARTD_PVT_ADS
>>> 01/08/18 10:03:26 Number of Active Workers 0
>>> 01/08/18 10:03:26 Number of Active Workers 0
>>> 01/08/18 10:03:26 DaemonCore: Can't receive command request from
>> xxx.xxx.xxx.105 (perhaps a timeout?)
>>
>> xxx.60 is one of my submit nodes, and xxx.105 is the central manager.
>>
>> There is also a similar entry for other nodes. I looked through for logs
>> with a bit more detail and got:
>>
>>> 01/08/18 09:59:56 DaemonCore: Can't receive command request from
>> xxx.xxx.xxx.105 (perhaps a timeout?)
>>> 01/08/18 09:59:56 PERMISSION DENIED to condor_pool@xxxxxxxxxxxxxxxxxxx
>> from host xxx.xxx.xxx.52 for command 10 (QUERY_STARTD_PVT_ADS), access
>> level NEGOTIATOR: reason: cached result for NEGOTIATOR; see first case
>> for the full reason
>>> 01/08/18 09:59:56 DC_AUTHENTICATE: Command not authorized, done!
>> Thank you for any further insight you can provide!
>> -Brandon
>>
>>
>> On 1/8/18 9:46 AM, John M Knoeller wrote:
>>> I think this means that condor_q is unable to fetch schedd ads from the collector.   
>>>
>>> Try running 
>>>
>>>    condor_status -schedd
>>>
>>> do you get the same error?
>>>
>>> does a simple 
>>>
>>>    condor_status 
>>>
>>> work?
>>>
>>> If you look in the CollectorLog on the central manager, do you see any messages about the rejected query?
>>>
>>> -----Original Message-----
>>> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Brandon Graves
>>> Sent: Monday, January 8, 2018 11:26 AM
>>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>>> Subject: [HTCondor-users] Error with global Queue
>>>
>>> Hello All,
>>>
>>> I recently replaced my Central Manager, and a few odd things have come
>>> up. The only definite error message I can find though happens when
>>> "condor_q -global" is run:
>>>
>>>> -- Failed to fetch ads from:
>>> <xxx.xxx.xxx.49:9618?addrs=xxx.xxx.xxx.49-9618+[> : server1.my.domain.com
>>>> AUTHENTICATE:1003:Failed to authenticate with any method
>>>> AUTHENTICATE:1004:Failed to authenticate using GSI
>>>> GSI:5003:Failed to authenticate.? Globus is reporting error
>>> (851968:50).? There is probably a problem with your credentials.? (Did
>>> you run grid-proxy-init?)
>>>> AUTHENTICATE:1004:Failed to authenticate using KERBEROS
>>>> AUTHENTICATE:1004:Failed to authenticate using FS
>>> My basic configuration is Central manager, connected to 2 submit nodes.
>>> Each submit node seems to be able to see it's own queue, one of the
>>> submit nodes off and on seems to be having trouble running jobs, but I
>>> can't seem to find any errors that make sense. For now I'd like to
>>> figure out the global queue error as I suspect they are related.
>>>
>>> My config file as far as authentication goes looks like this:
>>>
>>>
>>>> SEC_PASSWORD_FILE = /etc/condor/pool_password
>>>> SEC_DAEMON_AUTHENTICATION = REQUIRED
>>>> SEC_DAEMON_INTEGRITY = REQUIRED
>>>> SEC_DAEMON_AUTHENTICATION_METHODS = PASSWORD
>>>> SEC_NEGOTIATOR_AUTHENTICATION = REQUIRED
>>>> SEC_NEGOTIATOR_INTEGRITY = REQUIRED
>>>> SEC_NEGOTIATOR_AUTHENTICATION_METHODS = PASSWORD
>>>> SEC_CLIENT_AUTHENTICATION_METHODS = FS, PASSWORD, KERBEROS, GSI
>>> ( I didn't do the initial install/configuration of HTcondor on these
>>> systems, I'm just the new admin for them, and still getting my footing)
>>>
>>> I've looked through some of the logs, but I can't seem to find any
>>> specific error messages that point me in a new direction. Any
>>> tips/tricks/idea's would be appreciated
>>>
>>>
>>> --Brandon
>>>
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>>
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www-auth.cs.wisc.edu/lists/htcondor-users/attachments/20180109/88b66ada/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
HTCondor-users mailing list
HTCondor-users@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

------------------------------

End of HTCondor-users Digest, Vol 50, Issue 26
**********************************************