HTC System issues over the weekend

[chtc-users@xxxxxxxxxxx]

Dear CHTC users, 

 

This message is for users of our HTC system. 

 

Over the weekend, certain components of our large data file systems went down. This resulted in not being able to access files in /staging and /projects over the weekend. 

 

Both /staging and /projects folders should be back up and accessible. Because jobs using /staging or /projects likely failed or were stalled due to the weekend outage, we recommend inspecting these jobs if they ran or completed over the weekend.  

 

Separately, during the same time window, some users saw this error when running “condor_submit”: 

• “ERROR: store_cred of LOCAL credential failed”

This is a transient issue that should also now be resolved. We will continue investigating to determine if this error was related to the file system outage. 

 

If you have any ongoing problems with a /staging or /projects folder, or see the error message shown above, please contact us at chtc@xxxxxxxxxxx. 

 

Best, 

The CHTC Team