Re: [HTCondor-users] OAuthCredmon and Schedd: markfile names consistency

[Benoit Roland <benoit.roland@xxxxxxx>]

    Hi Jason,
    
    a big thanks a lot for your answer!
    
    I looked at the credmon interface [1] called from the schedd [2].
    All seems fine.
    
    o) Adding some log in [1], I got:
    
    SchedLog:09/11/23 05:27:10 (pid:4025430) (D_ALWAYS)Â CREDMON
    INTERFACE: cred_dir: /var/lib/condor/mytoken_credentials 
    SchedLog:09/11/23 05:27:10 (pid:4025430) (D_ALWAYS)Â CREDMON
    INTERFACE: user: benoit_roland 
    SchedLog:09/11/23 05:27:10 (pid:4025430) (D_ALWAYS)Â CREDMON
    INTERFACE: markfilename:
    /var/lib/condor/mytoken_credentials/benoit_roland.mark 
    
    o) Also, I set the parameter "SEC_CREDENTIAL_SWEEP_DELAY" from its
    default value of 1 week:
    ÂÂÂ SEC_CREDENTIAL_SWEEP_DELAY = 604800
    
    ÂÂÂ to 10 minutes:
    ÂÂÂ SEC_CREDENTIAL_SWEEP_DELAY = 600
    
    ÂÂ and the credentials were swept properly.
    
    ÂÂ Before sweeping: 
    ÂÂ 
    ÂÂ ls /var/lib/condor/mytoken_credentials/
    ÂÂ 
    ÂÂ b13941d2ba67c762d269fc4242980f74901b26f476747f90c48c82b5f6b1935cÂ
    
    ÂÂ benoit_roland.mark
    ÂÂ benoit_roland/
    
    ÂÂ After sweeping:
    ÂÂ 
    ÂÂ ls /var/lib/condor/mytoken_credentials/
    ÂÂ 
    ÂÂ b13941d2ba67c762d269fc4242980f74901b26f476747f90c48c82b5f6b1935cÂ
    
    ÂÂ benoit_roland.mark
    
    o) In the "OAuthCredmon.py", the threshold for deletion is given by
    "CREDMON_OAUTH_TOKEN_LIFETIME" if this parameter is defined.
    ÂÂÂ If not defined, the threshold is set to 1 day.
    
    ÂÂÂ Could we think to put a threshold value based on the actual job
    status, rather than a value defined once for all in the
    configuration file?
    ÂÂÂ For example, by looking if the access token has expired.
    
    ÂÂÂ If the user job is completed, the access token is not renewed
    anymore and expires within a short period of time (4000s in my
    case).
    ÂÂÂ The presence of an access token with a "time since creation"
    > 4000s - in my case - therefore means that the user job is
    completed
    ÂÂÂ and that the sweeping process can take place at that time.
    Â 
    Âo) Also, should we think to sweep the files of the kind
    "b13941d2ba67c762d269fc4242980f74901b26f476747f90c48c82b5f6b1935c"
    containing the information 
    ÂÂÂÂ about the oidc provider (Audience, AuthorizationUrl, ClientId,
    ClientSecret, ...)?
    ÂÂ Â If not swept when the jobs are finalized, these files pile up
    in the credential directory.
    
    Thanks a lot in advance for your feedback!
    
    Cheers,
    Benoit
    ÂÂÂ 
    [1] src/condor_utils/credmon_interface.cppÂÂ 
    [2] src/condor_schedd.V6/schedd.cpp  
    Â

On 08/09/2023 22:32, Jason Patton via HTCondor-users wrote:

Hi Benoit,

I believe you are correct about this code not looking at the correct path for the mark file. The good news is that this code is redundant, the condor_credd is in charge of sweeping the credentials (credmon_sweep_creds is the function referenced in the condor_credd code that handles this, if you're curious). I will plan on removing this code from the OAuth credmon...

...unless you are indeed finding that credentials are not being properly swept by the credd. In which case, it would be easy to fix the redundant code, but would also imply a bug in the credd that we should also look into.

Thanks for the detailed report!

Jason

On Thu, Sep 7, 2023 at 11:11âAM Benoit Roland <benoit.roland@xxxxxxx> wrote:

Dear all,

I understand that a markfile is used to take the decision to remove tokens when the markfile lifetime exceeds some threshold.

o) This is used in the functions "should_delete" and "delete_tokens" in the OAuth Credential Monitor [1].

In these functions, the markfile name is defined as:

mark_path = os.path.join(self.cred_dir, username, token_name + '.mark') [convention 1]

following the same convention as for the ".top" and ".use" tokens.

o) The markfile, if I am correct, is created in the schedd [2] via:

auto_free_ptr cred_dir_oauth(param("SEC_CREDENTIAL_DIRECTORY_OAUTH"));
....
credmon_mark_creds_for_sweeping(cred_dir_oauth, owner_info.Name());Â [convention 2]

with "credmon_mark_creds_for_sweeping(const char * cred_dir, const char* user)" defined in the credmon interface [3].

o) The first convention will result in e.g.:

SEC_CREDENTIAL_DIRECTORY_OAUTH/username/token_name.mark

while the second convention will result in:

SEC_CREDENTIAL_DIRECTORY_OAUTH/username.mark

o) Making a test, I can indeed find on my submit node:

/var/lib/condor/mytoken_credentials/benoit_roland.mark

while the OAuth Credential Monitor is looking for:

/var/lib/condor/mytoken_credentials/benoit_roland/helmholtz.mark

As a result, the markfile is not found, and the ".top", ".use" and ".mark" files are not deleted after expiration of the markfile.

Am I missing something, or are my observations correct?

Thanks a lot in advance for your help!

Cheers,
Benoit

[1] src/condor_credd/condor_credmon_oauth/credmon/CredentialMonitors/OAuthCredmon.py
[2] src/condor_schedd.V6/schedd.cpp
[3] src/condor_utils/credmon_interface.cpp

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/