[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] virtualenv, pip and htcondor (python)

Hi Brian & all,

Thanks for the help so far, I think I am almost there.
I've written the setup.py route off as a failure, as I cannot build older HTCondor versions (even after applying the patches).
Instead, I decided to try out `condaÂbuild` which turned out more successful so far.

I can now install htcondor python bindings on Ubuntu and SL6 under python 2.7 following the instructions in [1] (please give it a try).
However, a few things are missing/out of place:

  1. probably worth adding more tests if the bindings are working (currently just testing import of htcondor and classad)
  2. Although boost 1.63 is present & found by cmake, condor still downloads 1.49
  3. lots of binaries are compiled as well (and installed) - ideally none of this is needed. In addition to preventing themÂfrom overwriting the system ones (condaÂenv will be earlier in PATH), I would like to reduce the current size of 51 MB to around 20 MB if possible.
  4. condor is installing the python libraries into $PREFIX/lib/python instead of $PREFIX/lib/python2.7 (or $PREFIX/lib/python3.5)

The build file can be found in [2], the full build log in [3] and the currently available builds in [4].

This was an interesting side-project, now it is time to adjust the Luigi PR :).






On 3 January 2017 at 17:11, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

> On Jan 3, 2017, at 10:29 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:
> Hi Jaime,
> Thanks a lot.
> Finally got around to work a bit more on the setup.py.
> I've created a few docker images to be able to test the procedure in a more reliable manner:
> git clone https://github.com/kreczko/htcondor-1.git
> git checkout worlds-weirdest-setup.py
> cd docker
> # build containers
> docker-compose build htcondor-ubuntu
> docker-compose build htcondor-centos6
> docker-compose build htcondor-centos7
> # you can also try to directly install the modules:
> python setup.py install
> However, on my local machine the configuration fails for python, since cmake discovers two versions of python (2.7.11 & 3.5.2) and attempts to find headers for the latter only (for some reason I can find only 3.5.1 headers.
> Is there a way to point the configure script/cmake to the python version currently in use rather than the latest available?
> This is important for pip install with virtualenv or conda.

I think you can force a particular version by setting the following CMake variable:




> Am I considering all libraries (or too many)? Currently, I am copying
>Â - ./src/python-bindings/classad.so
>Â - ./src/python-bindings/htcondor.so
> to the python site-packages directory
> and
>Â - ./src/python-bindings/libpyclassad2.7_X_Y_X.so
>Â - ./src/condor_utils/libcondor_utils_X_Y_Z.so
> to /usr/lib (ubuntu) or /usr/lib64 (centos)

Those sound correct for the direct dependencies - did you use 'ldd' to suss out all the system dependencies?

> This one is also needed, but should already be present after a condor install:
>Â ./bld_external/classads-X.Y.Z/install/lib/libclassad.so

Well, it should actually look for something like "libclassad.so.7". The major version ("7") is relatively important because that should catch ABI breaks.


HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

 Dr Lukasz Kreczko     ÂÂ
 Research Associate
 Department of Physics
 Particle Physics Group

 University of Bristol

 HH Wills Physics Lab
 University of Bristol
 Tyndall Avenue
 BS8 1TL

 +44 (0)117 928 8724Â
 A top 5 UK university with leading employers (2015)
 A top 5 UK university for research (2014 REF)
 A world top 40 university (QS Ranking 2015)