Re: [HTCondor-users] HTCondor Personal Pool via Conda

[Jason Patton <jpatton@xxxxxxxxxxx>]

Hi Matt,

The HTCondor daemons (condor_master, condor_schedd, etc.) are not currently part of the conda-forge htcondor packages. Even with this constraint on the packages, constant maintenance is needed to keep them at/near the upstream version due to the patching that is needed to get the tools and bindings to build in the conda-forge environment, so there's pretty low appetite among the current maintainers to add more complexity to the packages. However, if anyone has time available to supply and maintain additional patches for building the daemons, the recipe is available here:

https://github.com/conda-forge/htcondor-feedstock

Otherwise, the htcondor/mini Docker image probably provides the best quickstart environment at the moment if you're not on a machine that already has HTCondor. If you find using htcondor.personal to start a pool inside a conda environment particularly useful, it will work if you install conda and the python-htcondor package in the htcondor/mini container:

>>> os.environ["PATH"] = f"/usr/sbin:{os.environ['PATH']}"
>>> pool.start()
PersonalPool(local_dir=~/personal-condor, state=READY)
>>> collector = pool.collector
>>> collector.query(htcondor.AdTypes.Any, projection=["MyType"])
[[ MyType = "Accounting" ], [ MyType = "Collector" ], [ MyType = "Machine" ], [ MyType = "Scheduler" ], [ MyType = "DaemonMaster" ], [ MyType = "Negotiator" ]]

(Note that adding /usr/sbin to the PATH will probably be needed in most cases.)

Jason Patton

On 9/1/22 10:25 AM, Matthew T West via HTCondor-users wrote:

While poking at the HTCondor install via Conda-forge today, I found that the personal pool startup faults out for lack of a condor_master.

>>> htcondor.version()
'$CondorVersion: 9.10.1 2022-07-19 $'
PersonalPool(local_dir=./personal-condor, state=INITIALIZED)
>>>
>>> pool.start()
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/mtw212/mambaforge/envs/htcondor/lib/python3.10/site-packages/htcondor/personal.py", line 112, in wrapper
ÂÂÂ return func(self, *args, **kwargs)
 File "/home/mtw212/mambaforge/envs/htcondor/lib/python3.10/site-packages/htcondor/personal.py", line 290, in start
ÂÂÂ self._start_condor()
 File "/home/mtw212/mambaforge/envs/htcondor/lib/python3.10/site-packages/htcondor/personal.py", line 112, in wrapper
ÂÂÂ return func(self, *args, **kwargs)
 File "/home/mtw212/mambaforge/envs/htcondor/lib/python3.10/site-packages/htcondor/personal.py", line 416, in _start_condor
ÂÂÂ self.condor_master = subprocess.Popen(
 File "/home/mtw212/mambaforge/envs/htcondor/lib/python3.10/subprocess.py", line 969, in __init__
ÂÂÂ self._execute_child(args, executable, preexec_fn, close_fds,
 File "/home/mtw212/mambaforge/envs/htcondor/lib/python3.10/subprocess.py", line 1845, in _execute_child
ÂÂÂ raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'condor_master'

While I know this copy of the codebase isn't a priority, it would be nice to have working. If this would work as a Conda install, it would be another easy way to get people up and running with a local system without much headache. I would be happy to test out changes to make sure stuff works.

Cheers,
Matt

-- 
Matthew T. West
DevOps & HPC SysAdmin
University of Exeter, Research IT
www.exeter.ac.uk/research/researchcomputing/support/researchit
57 Laver Building, North Park Road, Exeter, EX4 4QE, United Kingdom

Please note, I may send emails out of 'normal' working hours, as this fits my own work-life balance. I do not expect a response outside of your own working hours.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/