[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] virtualenv, pip and htcondor (python)



Hi Jaime,

Thanks a lot.

Finally got around to work a bit more on the setup.py.

I've created a few docker images to be able to test the procedure in a more reliable manner:
git cloneÂhttps://github.com/kreczko/htcondor-1.git
git checkoutÂworlds-weirdest-setup.py
cd docker
# build containers
docker-compose build htcondor-ubuntu
docker-compose build htcondor-centos6
docker-compose build htcondor-centos7

# you can also try to directly install the modules:
python setup.py install

However, on my local machine the configuration fails for python, since cmake discovers two versions of python (2.7.11 & 3.5.2) and attempts to find headers for the latter only (for some reason I can find only 3.5.1 headers.
Is there a way to point the configure script/cmakeÂto the python version currently in use rather than the latest available?
This is important for pip install with virtualenv or conda.

Am I considering all libraries (or too many)? Currently, I am copying
Â- ./src/python-bindings/classad.so
Â- ./src/python-bindings/htcondor.so
to the python site-packages directory
and
Â- ./src/python-bindings/libpyclassad2.7_X_Y_X.so
Â- ./src/condor_utils/libcondor_utils_X_Y_Z.so
to /usr/lib (ubuntu) or /usr/lib64 (centos)

This one is also needed, but should already be present after a condor install:
Â./bld_external/classads-X.Y.Z/install/lib/libclassad.so

Cheers,
Luke


On 16 December 2016 at 14:59, Jaime Frey <jfrey@xxxxxxxxxxx> wrote:
Iâve fixed the problem with WANT_KRB5=FALSE on our master branch.

Â- Jaime

On Dec 14, 2016, at 10:25 AM, John M Knoeller <johnkn@xxxxxxxxxxx> wrote:

This looks like a bug in our handling of WANT_KRB5. Iâm not sure when it will be patched
So you might want to add your own #ifdefs to condor_auth_ssl.cpp so you can move on to the next thing ;)
Â
-tj
Â
From:ÂHTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx]ÂOn Behalf OfÂL Kreczko
Sent:ÂSaturday, December 10, 2016 10:13 AM
To:ÂHTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject:ÂRe: [HTCondor-users] virtualenv, pip and htcondor (python)
Â
Hi John, hi Brian,
Â
Thanks for the answers. After a bit of fiddling I ended up with this set:
cmake -DPROPER:BOOL=FALSE -DCLIPPED:BOOL=TRUE -DWITH_BLAHP:BOOL=FALSE -DWITH_COREDUMPER:BOOL=FALSE \
   -DWITH_CREAM:BOOL=FALSE -DWITH_DRMAA:BOOL=FALSE -DWITH_GLOBUS:BOOL=FALSE -DWITH_GSOAP:BOOL=FALSE \
   -DWITH_HADOOP:BOOL=FALSE -DWITH_LIBVIRT:BOOL=FALSE -DWITH_LIBXML2:BOOL=FALSE -DWITH_UNICOREGAHP:BOOL=FALSE \
   -DWITH_VOMS:BOOL=FALSE -DWITH_BOINC:BOOL=FALSE -DWITH_KRB5:BOOL=FALSE -DWITH_LIBCGROUP:BOOL=OFF\
   -DWITH_GANGLIA:BOOL=OFF -DWITH_CAMPUSFACTORY:BOOL=OFF -DWITH_BOSCO:BOOL=OFF -DWANT_GLEXEC:BOOL=OFF \
   -DUW_BUILD:BOOL=TRUE -D_VERBOSE:BOOL=TRUE -DBUILDID:STRING=UW_development -DCMAKE_INSTALL_PREFIX:PATH=${PWD}/release_dir "$@"
Â
However, since I turn off KRB5 an error occurs:
/home/vagrant/workspace/htcondor/src/condor_io/condor_auth_ssl.cpp: In static member function âstatic bool Condor_Auth_SSL::Initialize()â:
/home/vagrant/workspace/htcondor/src/condor_io/condor_auth_ssl.cpp:107:7: error: âCondor_Auth_Kerberosâ has not been declared
 if ( Condor_Auth_Kerberos::Initialize() == false ||
   Â^
src/condor_utils/CMakeFiles/condor_utils.dir/build.make:7152: recipe for target 'src/condor_utils/CMakeFiles/condor_utils.dir/__/condor_io/condor_auth_ssl.cpp.o' failed.
Â
Â
If I allow for KRB, the compilation fails at DAG:
[ 74%] Linking CXX executable condor_submit_dag
[ 74%] Built target condor_submit_dag
Scanning dependencies of target condor_dagman_metrics_reporter
[ 74%] Building CXX object src/condor_dagman/CMakeFiles/condor_dagman_metrics_reporter.dir/condor_dagman_metrics_reporter.cpp.o
[ 74%] Building CXX object src/condor_dagman/CMakeFiles/condor_dagman_metrics_reporter.dir/__/condor_utils/condor_version.cpp.o
[ 74%] Linking CXX executable condor_dagman_metrics_reporter
//lib/x86_64-linux-gnu/libssl.so.1.0.0: undefined reference to `SHA224_Init@xxxxxxxxxxxxx'
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ENGINE_get_id@xxxxxxxxxxxxx'
+ many more 'undefined reference'.
Â
Is there a way to disable DAG? Essentially, I just want the python-bindings :)
Â
And just as I said that, IÂ
cd src/python-bindings/
make
Â
and end up with htcondor.so and classad.so!
Well, that was easy ;)
Â
Â
Starting from fresh, the python-bindings also compile
Â- src/classad
Â- src/condor_utils
Â
Â
Next step is to make the compiled libraries work with the installed ones (e.g import the module):
python -c 'import htcondor'Traceback (most recent call last):
 File "<string>", line 1, in <module>
ImportError: /home/vagrant/workspace/htcondor/src/python-bindings/libcondor_utils_8_5_9.so: undefined symbol: _ZN7classad19ClassAdJsonUnParser7UnparseERNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKNS_7ClassAdERKSt3setIS6_NS_12CaseIgnLTStrESaIS6_EE
Â
BTW: 'import classad' works out-of-the-box'
Â
For the setup.py there are two approaches:
Not experienced with either, suggestions are welcome.
Â
Â
Cheers,
Luke
Â
On 8 December 2016 at 18:21, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

> On Dec 8, 2016, at 7:29 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:
>
> Hi all,
>
> Since my work on an HTCondor task implementation for Spotify's Luigi [1], this topic is again becoming of interest.
> In order to do 'pip install htcondor', htcondor would need a setup.py that compiles the HTCondor Python bindings and distributes the libraries htcondor.so and classad.so the same way the condor-python RPM does.
>
> As I understand from the instructions [2] all I need to call in the setup.py is
> 1) match code to the installed condor version
> 2) ./configure_uw
> 3) make package
> 4) point python to the two libraries it needs to install.
>

I suspect you'll need to pick apart ./configure_uw and aggressively disable cmake flags. For example, the error you reference on 16.04 is due to a failure building a unit test. You can do without those!

> Is this a safe way? Is it possible to compile just the python bindings (I am asking because I cannot build condor on 16.04 [3])?
> Do htcondor.so and classad.so need to be compiled in the same version as the condor install or does some wiggle-room exist?
>

All variants of the compilation should result in a compatible protocol layer. However, if you disable some of the security libraries (think Globus, Kerberos), the security negotiation might fail.

Brian

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/



--
*********************************************************
 Dr Lukasz Kreczko     ÂÂ
 Research Associate
 Department of Physics
 Particle Physics Group

 University of Bristol

 HH Wills Physics Lab
 University of Bristol
 Tyndall Avenue
 Bristol
 BS8 1TL


 +44 (0)117 928 8724Â
ÂÂ
 A top 5 UK university with leading employers (2015)
 A top 5 UK university for research (2014 REF)
 A world top 40 university (QS Ranking 2015)
*********************************************************