0 *H 010  `He0 *H $PContent-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit I turned on the insane debugging and got the following: _________________________________________________________________________________________________________________ 06/23/20 09:56:46 (fd:13) (pid:5857) (D_ALWAYS) read_password_from_filename(): read_secure_file(/etc/condor/password.d/POOL) failed! 06/23/20 09:56:46 (fd:13) (pid:5857) (D_PRIV) PRIV_CONDOR --> PRIV_ROOT at /var/lib/condor/execute/slot3/dir_3977/userdir/.tmpEsbepJ/BUILD/condor-8.9.7/src/condor_utils/secure_file.cpp:213 06/23/20 09:56:46 (fd:13) (pid:5857) (D_PRIV) PRIV_ROOT --> PRIV_CONDOR at /var/lib/condor/execute/slot3/dir_3977/userdir/.tmpEsbepJ/BUILD/condor-8.9.7/src/condor_utils/secure_file.cpp:216 06/23/20 09:56:46 (fd:13) (pid:5857) (D_ALWAYS:2) ERROR: read_secure_file(/etc/condor/password.d/POOL): open() failed: No such file or directory (errno: 2) 06/23/20 09:56:46 (fd:13) (pid:5857) (D_ALWAYS) read_password_from_filename(): read_secure_file(/etc/condor/password.d/POOL) failed! 06/23/20 09:56:46 (fd:13) (pid:5857) (D_SECURITY) PW: Server sending. 06/23/20 09:56:46 (fd:13) (pid:5857) (D_SECURITY) In server_send: -1. 06/23/20 09:56:46 (fd:13) (pid:5857) (D_SECURITY) Server send '', '', 0 0 0 __________________________________________________________________________________________________________________ No symlinks in the hierarchy, everything owned by root:root with exec permissions all the way down. I believe I am starting the daemons as root? It looks like the privilege escalation is going fine based on what the log shows here. At least I have something to go off of now, but at first glance I don't understand why open() would fail with code 2 on that file. I double checked with a python interactive shell as root and I was able to open and spit out bytes from that file just fine. My thinking is that maybe I am not starting condor correctly, so it can't try to open the file as root? I am on CentOS7, and started condor with 'sudo systemctl start condor'. I also double-checked my security config, and the only differences between my config file and the readthedocs are: ______________________________ ALLOW_DAEMON = * ALLOW_NEGOTIATOR = * ______________________________ Which I do not think are relevant to this issue. Does anyone have any more ideas on things I could check to see what's going on? -Wes Wesley Taylor – Cluster Manager Numerica Corporation (www.numerica.us) 5042 Technology Parkway #100 Fort Collins, Colorado 80528 ☎️ (970) 207 2232 📧 wesley.taylor@numerica.us -----Original Message----- From: HTCondor-users On Behalf Of Zach Miller Sent: Tuesday, June 23, 2020 9:30 AM To: HTCondor-Users Mail List Subject: Re: [HTCondor-users] [External] - Re: Debugging HTCondor Authentication Errors CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hello, You might also check the permissions and ownership on the parent directories. read_secure_file() wants to make sure the file couldn't have been tampered with. Are there symlinks in the directory hierarchy? You are starting the daemons as root? You can turn on more (insane) debugging by setting "COLLECTOR_DEBUG = D_ALL:2". It is A LOT, but that will get you every message. You may also need to increase the size of the log if it rotates too quickly (MAX_COLLECTOR_LOG). Looking at the code, though, read_secure_file() should already be logging why it failed even without changing the debug level. Is there anything earlier in the log that gives a hint? (Feel free to send the whole log offline if you'd like). Cheers, -zach On 6/23/20, 10:20 AM, "HTCondor-users on behalf of wesley.taylor@numerica.us" wrote: Hi Zach! Thanks for that explanation, I was obviously looking in the wrong direction there. I changed the permissions like you said, but I am still getting the same error from my collector about reading the file. Is there a debug setting I can add to get it to tell me more about why the read_secure_file call failed? -Wes Wesley Taylor – Cluster Manager Numerica Corporation (https://usg02.safelinks.protection.office365.us/?url=http%3A%2F%2Fwww.numerica.us%2F&data=02%7C01%7C%7Cde9e15f0f6384ad5e08208d8178ab928%7Cfae7a2aedf1d444e91bebabb0900b9c2%7C0%7C0%7C637285231872717967&sdata=5XBBEpxWZ2gmgtKwtxy0ckfEkCFKyqo7EbkP13MRhMw%3D&reserved=0) 5042 Technology Parkway #100 Fort Collins, Colorado 80528 ☎️ (970) 207 2232 📧 wesley.taylor@numerica.us -----Original Message----- From: HTCondor-users On Behalf Of Zach Miller Sent: Monday, June 22, 2020 10:02 PM To: HTCondor-Users Mail List Subject: [External] - Re: [HTCondor-users] Debugging HTCondor Authentication Errors CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Wes, (First, no problem sending email any time you have a question!) You wrote: > It looks like for some reason condor can't read the POOL file, even > though the file (and its parent directory) are owned by the user and > group condor:condor, and everyone has execute permissions on /etc and > /etc/condor. I also made selinux permissive just in case that was the > issue. Clearly the error message here should be better. When HTCondor says, "read_secure_file(/etc/condor/password.d/POOL) failed!", in this case it's not because it couldn't read the file but that the file was TOO permissive. The file for PASSWORD authentication should be chmod 600 and owned by root:root. Try fixing that and let us know if that did the trick. Cheers, -zach On 6/22/20, 7:11 PM, "HTCondor-users on behalf of wesley.taylor@numerica.us" wrote: Hello, I feel a little bad for emailing everyone twice in the same day, but I am still getting familiar with HTCondor. I tested a minicondor last week and was having a ball of a time, but now I am looking to scale up and have hit some hiccups. I am following the "Setting up an HTCondor Pool" (https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fhtcondor.readthedocs.io%2Fen%2Flatest%2Fadmin-manual%2Fquick-start-condor-pool.html&data=02%7C01%7C%7Cde9e15f0f6384ad5e08208d8178ab928%7Cfae7a2aedf1d444e91bebabb0900b9c2%7C0%7C0%7C637285231872717967&sdata=YX2xTyQwlSlmcAdvjcvKig%2BsyHxijnD%2BVE0d4QWWMhI%3D&reserved=0) but with some minor modifications to try and make the test setup better match the production system's architecture. I simply changed the configuration so one machine had the roles of "Submit" and "Central Manager" and I have two "Execute" machines located on the same network. I went through the guide, but when I started everything up they weren't authenticating with one another. On both the "Execute" machines I am getting the following from my StartLog (I set STARTD_DEBUG = D_SECURITY:2 in my config): __________________________________________________________________________________________________________________________________________________________________________________________________ 06/22/20 17:33:14 SECMAN: new session, doing initial authentication. 06/22/20 17:33:14 SECMAN: authenticating RIGHT NOW. 06/22/20 17:33:14 SECMAN: AuthMethodsList: PASSWORD 06/22/20 17:33:14 SECMAN: Auth methods: PASSWORD 06/22/20 17:33:14 AUTHENTICATE: setting timeout for <192.168.0.69:9618> to 20. 06/22/20 17:33:14 AUTHENTICATE: in authenticate( addr == '<192.168.0.69:9618>', methods == 'PASSWORD') 06/22/20 17:33:14 AUTHENTICATE: can still try these methods: PASSWORD 06/22/20 17:33:14 HANDSHAKE: in handshake(my_methods = 'PASSWORD') 06/22/20 17:33:14 HANDSHAKE: handshake() - i am the client 06/22/20 17:33:14 HANDSHAKE: sending (methods == 512) to server 06/22/20 17:33:14 HANDSHAKE: server replied (method = 512) 06/22/20 17:33:14 AUTHENTICATE: will try to use 512 (PASSWORD) 06/22/20 17:33:14 AUTHENTICATE: do_authenticate is 1. 06/22/20 17:33:14 PW. 06/22/20 17:33:14 PW: getting name. 06/22/20 17:33:14 PW: Generating ra. 06/22/20 17:33:14 PW: Client sending. 06/22/20 17:33:14 Client sending: 0, 19(condor_pool@worker1), 256 06/22/20 17:33:14 PW: Client receiving. 06/22/20 17:33:14 Server sent status indicating not OK. 06/22/20 17:33:14 PW: Client received ERROR from server, propagating 06/22/20 17:33:14 PW: CLient sending two. 06/22/20 17:33:14 In client_send_two. 06/22/20 17:33:14 Can't send null for random string. 06/22/20 17:33:14 Client sending: 0() 0 0 06/22/20 17:33:14 Sent ok. 06/22/20 17:33:14 AUTHENTICATE: method 512 (PASSWORD) failed. 06/22/20 17:33:14 AUTHENTICATE: can still try these methods: 06/22/20 17:33:14 HANDSHAKE: in handshake(my_methods = '') 06/22/20 17:33:14 HANDSHAKE: handshake() - i am the client 06/22/20 17:33:14 HANDSHAKE: sending (methods == 0) to server 06/22/20 17:33:14 HANDSHAKE: server replied (method = 0) 06/22/20 17:33:14 AUTHENTICATE: no available authentication methods succeeded! 06/22/20 17:33:14 SECMAN: required authentication with collector 192.168.0.69 failed, so aborting command DC_START_TOKEN_REQUEST. 06/22/20 17:33:14 Failed to request a new token: DAEMON:1:failed to start command for token request with remote daemon at '<192.168.0.69:9618>'.|AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using PASSWORD __________________________________________________________________________________________________________________________________________________________________________________________________ So then I went and looked at the CollectorLog on the Manager: __________________________________________________________________________________________________________________________________________________________________________________________________ 06/22/20 18:03:15 DC_AUTHENTICATE: required authentication of 192.168.0.70 failed: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using PASSWORD 06/22/20 18:03:15 read_password_from_filename(): read_secure_file(/etc/condor/password.d/POOL) failed! 06/22/20 18:03:15 read_password_from_filename(): read_secure_file(/etc/condor/password.d/POOL) failed! ___________________________________________________________________________________________________________________________________________________________________________________________________ (Don't pay attention to the fact the timestamps are really far apart, I have just been trying some more things in the past little bit) It looks like for some reason condor can't read the POOL file, even though the file (and its parent directory) are owned by the user and group condor:condor, and everyone has execute permissions on /etc and /etc/condor. I also made selinux permissive just in case that was the issue. Does anyone have any further steps I can take to figure out why this read is failing? Thank you! -Wes Wesley Taylor – Cluster Manager Numerica Corporation (https://usg02.safelinks.protection.office365.us/?url=http%3A%2F%2Fwww.numerica.us%2F&data=02%7C01%7C%7Cde9e15f0f6384ad5e08208d8178ab928%7Cfae7a2aedf1d444e91bebabb0900b9c2%7C0%7C0%7C637285231872717967&sdata=5XBBEpxWZ2gmgtKwtxy0ckfEkCFKyqo7EbkP13MRhMw%3D&reserved=0) 5042 Technology Parkway #100 Fort Collins, Colorado 80528 ☎️ (970) 207 2232 📧 wesley.taylor@numerica.us _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a subject: Unsubscribe You can also unsubscribe by visiting https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Flists.cs.wisc.edu%2Fmailman%2Flistinfo%2Fhtcondor-users&data=02%7C01%7C%7Cde9e15f0f6384ad5e08208d8178ab928%7Cfae7a2aedf1d444e91bebabb0900b9c2%7C0%7C0%7C637285231872717967&sdata=1kQ1hN%2FIorjiWT7JRHXPodEiLznW6ycKXgqUH%2B9tOHo%3D&reserved=0 The archives can be found at: https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Flists.cs.wisc.edu%2Farchive%2Fhtcondor-users%2F&data=02%7C01%7C%7Cde9e15f0f6384ad5e08208d8178ab928%7Cfae7a2aedf1d444e91bebabb0900b9c2%7C0%7C0%7C637285231872717967&sdata=Wh6oeUE1WLnH2MTYVV8ND2u3uB%2F1i9wUyI8DIxKf%2FIE%3D&reserved=0 _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a subject: Unsubscribe You can also unsubscribe by visiting https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Flists.cs.wisc.edu%2Fmailman%2Flistinfo%2Fhtcondor-users&data=02%7C01%7C%7Cde9e15f0f6384ad5e08208d8178ab928%7Cfae7a2aedf1d444e91bebabb0900b9c2%7C0%7C0%7C637285231872717967&sdata=1kQ1hN%2FIorjiWT7JRHXPodEiLznW6ycKXgqUH%2B9tOHo%3D&reserved=0 The archives can be found at: https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Flists.cs.wisc.edu%2Farchive%2Fhtcondor-users%2F&data=02%7C01%7C%7Cde9e15f0f6384ad5e08208d8178ab928%7Cfae7a2aedf1d444e91bebabb0900b9c2%7C0%7C0%7C637285231872717967&sdata=Wh6oeUE1WLnH2MTYVV8ND2u3uB%2F1i9wUyI8DIxKf%2FIE%3D&reserved=0 007OKH0  *H  0[10 &,dus10 &,dnumerica10 &,dad10Uad-GILGALAD-CA0 160724195713Z 360724200712Z0[10 &,dus10 &,dnumerica10 &,dad10Uad-GILGALAD-CA0"0  *H 0 ws%ƺNƮE Pʵpcٰ%(vC%?ۦj[.mKStRq;Y7J#p5[PxV5KL(X\jEJðkI8g{ëyMgBZrusQ8?79p{wTddvfGBO]a U@Ԡ@ҏ @<p\2-pZY=$7~/#\6 Ad.A&-6CNXD Wb 6W`'q.K1 d{go^@ iZ1 !Hͻv(|aomi%ϨbPc3f嚻]l;p]Bx<~\eK y˻*C*RqhK^9h֐]D^xѢ>=Y\#9g=dDq&qd] )0?fk!܎5e$J Ty ˬ ;i0g0 +7CA0U0U00U1fOzTRz!{.0 +70  *H  '0g!Tcᡇm wa&y<09/fKmWܠ x$db@MZV'3G^8 UsdSE7?Hn؂t_(E+w룓[F_%G7CQTaaB ^Tf%,oYy MQ9"$:eX(Q́qda6a3ؖ"x>cم/F)zznP6-w3{l0y ̸KNHez"=A ` F .vBS5$Q&q)=lH&CypBY iԶd rlK/ȉJ%y)]'˖e؞W~L:8sGaC†iQ2@yƑ?71RKU m1!00s0\10 &,dus10 &,dnumerica10 &,dad10Uad-CELEBRIAN-CA' 16F;GdU 0  `He0 *H  1  *H 0 *H  1 200623165619Z0O *H  1B@G~Q_]K ^Xؖ%(r:n9z&^]Iyr(伎.QGɎ*|0R *H  1C0A |ixgJ F]00wesley.taylor@numerica.us0 +71u0s0\10 &,dus10 &,dnumerica10 &,dad10Uad-CELEBRIAN-CA' 16F;GdU 0 *H   1us0\10 &,dus10 &,dnumerica10 &,dad10Uad-CELEBRIAN-CA' 16F;GdU 0 *H  100  `He*0  `He0 *H 0  `He0*H 0 *H @0  `He0  `He0  `He0+0  *H 2 hK-c±NiM7:[A\-R 9K ǔA q hTGc8)*6Jy66G 󅕚RD {Q(W!Rk^ }FS_(Z煜 7bOZG nx5S[|5agTO& >gl|JvQQw$!45,E֌v}=ad,{