Re: [HTCondor-users] Have submit nodes inside a Kubernetes cluster

[Miguel Garrido <miguel@xxxxxxxxx>]

Hi -

172.18.0.1 looks like a Docker bridge network address which is likely not reachable from another host on the network e.g. the CM. This shouldnât be a problem within the Kubernetes cluster since each pod should be able to route to another one within the cluster (with proper configs).

You could try in your CentOS test case to have your container use a bridge that assigns the container an externally reachable address from the host systemâs network.Â

RegardsÂ

On Tue, May 2, 2023 at 09:09 Gaetan Geffroy <gage@xxxxxxxxx> wrote:

Hi,

Â

I know it is possible to run condor entirely inside Kubernetes.

But what if I have an existing pool outside of the Kubernetes cluster, with the central manager and the executor nodes, could I spawn one or several pods in the Kubernetes cluster and have them act as submit nodes ?

Â

As a quick test I tried the following:

• Have a pool running on different machines (one CM and two nodes)
• On another machine, not a member of the pool but part of the same network, I start a Docker in Docker container
• Inside this dind container, I start a CentOS container , on which I run âcurl -fsSL https://get.htcondor.org | GET_HTCONDOR_PASSWORD="some_secret_password" /bin/bash -s -- --no-dry-run --submit my.cm.adressâ

After doing this, from inside the CentOS container I can use condor_status fine, but when I submit jobs they stay IDLE forever and the negotiator log says something like:

Â

Failed to connect to condor@xxxxxxxxxxxxxxxxxx (<172.18.0.1:9618?addrs=172.18.0.1-9618&alias=condor.test.submit&noUDP&sock=schedd_412_7d83>)

Error: Ignoring submitter for this cycle

Â

So it looks like the CM has trouble connecting back to the submit container.

I tried the same setup but outside of the dind container (spawning the CentOS container directly on the docker host machine) and there it works well.

Â

So, seeing that, I am starting to have my doubts about how having Kubernetes pods connecting to the pool and submitting jobs.

Was this ever done before ? Did it work ? Could it work ?

Â

Thanks,

GaÃtan

Â

Gaetan Geffroy
Junior Software Engineer,ÂSpace

Terma GmbH
Europaarkaden II, BratustraÃe 7, 64293 Darmstadt, Germany
TÂ+49 6151 86005 43Â(direct)ÂÂâÂÂTÂ+49 6151 86005-0
Terma GmbHÂ-ÂSitz DarmstadtÂÂâÂÂHandelsregisterÂNr.:ÂHRBÂ7411,ÂDarmstadt
GeschÃftsfÃhrer:ÂPoul VighÂ/ÂSteen Vejby SÃrensen
www.terma.comÂâÂLinkedin â Twitter â Instagram â Youtube

---

Attention:
This e-mail (and attachment(s), if any) - intended for the addressee(s) only - may contain confidential, copyright, or legally privileged information or material, and no one else is authorized to read, print, store, copy, forward, or otherwise use or disclose any part of its contents or attachment(s) in any form. If you have received this e-mail in error, please notify me by telephone or return e-mail, and delete this e-mail and attachment(s). Thank you.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

MG