Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] htcondor python bindings for python3 ?

Date: Tue, 26 Jan 2016 12:08:31 +0200
From: Aki Ketolainen <akik@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] htcondor python bindings for python3 ?

Hi,

I've managed to compile htcondor 8.4.3 (commit8248906d6ce2b5cc9ad998393b3c1db7725768e4) with

python 3.4.4 and boost 1.54.0 to have the python3 python-bindings.

I'm not a programmer but a friend who is, helped with the errors thatcame up.I'll list the changes as diffs in this e-mail so that you could possiblycomment whether

I broke something along the way (very probable :)

python 3.4.4:

LDFLAGS=-Wl,-rpath=/sw/python/3.4.4/lib,--enable-new-dtags./configure --prefix=/sw/python/3.4.4 --enable-shared && make && makeinstall


boost-1.54.0:
    export CPATH=/sw/python/3.4.4/include/python3.4m

./bootstrap.sh --prefix=/sw/python/3.4.4 --with-python=python3--with-python-root=/sw/python/3.4.4 && ./b2 && ./b2 install


htcondor-8.4.3:

--- htcondor/build/cmake/CondorConfigure.cmake 2016-01-2609:20:42.380096304 +0000+++ htcondor.python3/build/cmake/CondorConfigure.cmake 2016-01-2522:49:16.788096304 +0000

@@ -782,7 +782,7 @@

   add_subdirectory(${CONDOR_EXTERNAL_DIR}/bundles/drmaa/1.6.2)
   add_subdirectory(${CONDOR_EXTERNAL_DIR}/bundles/qpid/0.8-RC3)
-  add_subdirectory(${CONDOR_EXTERNAL_DIR}/bundles/boost/1.49.0)
+  add_subdirectory(${CONDOR_EXTERNAL_DIR}/bundles/boost/1.54.0)

   add_subdirectory(${CONDOR_EXTERNAL_DIR}/bundles/curl/7.31.0-p1 )
   add_subdirectory(${CONDOR_EXTERNAL_DIR}/bundles/openssl/1.0.1e)

--- htcondor/CMakeLists.txt    2016-01-26 09:21:01.451096304 +0000

+++ htcondor.python3/CMakeLists.txt 2016-01-25 22:49:32.351096308+0000

@@ -16,6 +16,15 @@
  #
  ###############################################################

+set(PYTHON_LIBRARY "/sw/python/3.4.4/lib/libpython3.so")
+set(PYTHON_INCLUDE_DIR "/sw/python/3.4.4/include/python3.4m")
+set(PYTHON_EXECUTABLE "/sw/python/3.4.4/bin/python3")
+set(PYTHONLIBS_VERSION_STRING "3.4.4")
+set(Python_ADDITIONAL_VERSIONS "3.4.4")
+set(PYTHON_VERSION_MAJOR "3")
+set(PYTHON_VERSION_MINOR "4")
+set(PYTHON_VERSION_PATCH "4")
+

##########################################################################

 # Set project
 project(CONDOR)

--- htcondor/externals/bundles/boost/1.54.0/CMakeLists.txt2016-01-26 09:20:42.398096303 +0000+++ htcondor.python3/externals/bundles/boost/1.54.0/CMakeLists.txt2016-01-25 22:49:52.260096280 +0000

@@ -16,6 +16,19 @@
  #
  ###############################################################

+set(BOOST_ROOT /sw/boost/1.54.0)
+set(BOOST_INCLUDEDIR /sw/boost/1.54.0/include)
+set(BOOST_INCLUDE /sw/boost/1.54.0/include)
+set(BOOST_LIBRARYDIR /sw/boost/1.54.0/lib)
+set(Boost_NO_SYSTEM_PATHS ON)
+set(Boost_FOUND TRUE)
+set(Boost_INCLUDE_DIRS /sw/boost/1.54.0/include)
+set(Boost_LIBRARY_DIRS /sw/boost/1.54.0/lib)
+set(HAVE_EXT_BOOST TRUE)
+set(BOOST_VER "1.54.0")
+set(BOOST_LD /sw/boost/1.54.0/lib)
+set(Boost_PYTHON_LIBRARY /sw/boost/1.54.0/lib/libboost_python3.so)
+
 # Search for system's boost libraries.
 condor_pre_external( BOOST boost-1.54.0 "lib;boost" "done")

--- htcondor/src/classad/tests/withcache.cpp 2016-01-2609:20:42.703096303 +0000+++ htcondor.python3/src/classad/tests/withcache.cpp 2016-01-2522:50:35.379096298 +0000

@@ -1,3 +1,4 @@
+#define BOOST_TEST_DYN_LINK
 /***************************************************************
  *
  * Copyright (C) 1990-2012, Red Hat Inc.

--- htcondor/src/classad/tests/without_cache.cpp 2016-01-2609:20:42.703096303 +0000+++ htcondor.python3/src/classad/tests/without_cache.cpp2016-01-25 22:50:41.924096318 +0000

@@ -1,3 +1,4 @@
+#define BOOST_TEST_DYN_LINK
 /***************************************************************
  *
  * Copyright (C) 1990-2012, Red Hat Inc.

--- htcondor/src/condor_utils/tests/ring_buffer_tests.cpp2016-01-26 09:20:42.993096304 +0000+++ htcondor.python3/src/condor_utils/tests/ring_buffer_tests.cpp2016-01-25 22:50:57.321096305 +0000

@@ -1,3 +1,4 @@
+#define BOOST_TEST_DYN_LINK
 /***************************************************************
  *
  * Copyright (C) 1990-2013, Red Hat Inc.

--- htcondor/src/condor_collector.V6/tests/collector_tests.cpp2016-01-26 09:20:42.909096304 +0000+++ htcondor.python3/src/condor_collector.V6/tests/collector_tests.cpp2016-01-25 22:51:12.800096307 +0000

@@ -1,3 +1,4 @@
+#define BOOST_TEST_DYN_LINK
 /***************************************************************
  *
  * Copyright (C) 1990-2012, Red Hat Inc.

--- htcondor/src/python-bindings/classad_module.cpp 2016-01-2609:20:42.918096304 +0000+++ htcondor.python3/src/python-bindings/classad_module.cpp2016-01-25 22:51:53.206096285 +0000

@@ -71,7 +71,7 @@
     if (fd == -1)
     {
         PyErr_Clear();
-        return nullptr;
+        return 0;
     }
     int flags = fcntl(fd, F_GETFL);
     if (flags == -1)

--- htcondor/src/python-bindings/classad_python_user.cpp2016-01-26 09:20:42.914096304 +0000+++ htcondor.python3/src/python-bindings/classad_python_user.cpp2016-01-25 22:52:16.005096314 +0000

@@ -48,7 +48,7 @@
     {
         if (!Py_IsInitialized())
         {
-            char pname[] = "htcondor";
+            wchar_t pname[] = L"htcondor";
             Py_SetProgramName(pname);
             Py_InitializeEx(0);
         }

--- htcondor/src/python-bindings/collector.cpp 2016-01-2609:21:01.537096304 +0000+++ htcondor.python3/src/python-bindings/collector.cpp 2016-01-2522:52:35.772096309 +0000

@@ -80,7 +80,7 @@
             m_collectors = CollectorList::create();
             m_default = true;
         }

- else if (PyString_Check(pool.ptr()) ||PyUnicode_Check(pool.ptr()))+ else if (PyBytes_Check(pool.ptr()) ||PyUnicode_Check(pool.ptr()))

std::string pool_str =boost::python::extract<std::string>(pool);

             if (pool_str.size())

--- htcondor/configure_redhat  2016-01-26 09:20:42.077096304 +0000

+++ htcondor.python3/configure_redhat 2016-01-25 22:48:45.048096317+0000

@@ -27,4 +27,11 @@
   -DBUILDID:STRING=RH_development \
   -DWITH_GLOBUS:BOOL=FALSE \
   -DWITH_VOMS:BOOL=FALSE \
+  -DWITH_CREAM:BOOL=FALSE \
+  -DWITH_BLAHP:BOOL=FALSE \
+  -DPROPER:BOOL='OFF' -D_VERBOSE:BOOL='ON' \
+  -DWITH_PYTHON_BINDINGS:BOOL=TRUE \
+  -DCLIPPED:BOOL=FALSE \
+  -DNO_PHONE_HOME:BOOL=TRUE \
   -DCMAKE_INSTALL_PREFIX:PATH=${PWD}/release_dir "$@"
+

Then ./configure_redhat and make.
Thank you for the support,

Aki


On 2016-01-21 12:16, Aki Ketolainen wrote:

Hi,

(Sorry for this second message, the first one was rejected.)

I've tried to "UW" compile htcondor from source but I haven't been
able to complete the compile and
produce lib/python/htcondor.so and lib/python/classad.so so that
python3 would be able to use them.

If I set the cmake variables for the python3 version in htcondor main
CMakeLists.txt

set(PYTHON_LIBRARY "/sw/python/3.4.4/lib/libpython3.so")
set(PYTHON_INCLUDE_DIR "/sw/python/3.4.4/include/python3.4m")
set(PYTHON_EXECUTABLE "/sw/python/3.4.4/bin/python3.4m")

and have python3 in my path, boost compile fails. By default the cmake
compile tries to compile

boost 1.49.0 and according to you, boost 1.49.0 is not compatible withpython3.


So I'd need to find a way to have the cmake process try to use boost
1.54.0 instead of 1.49.0 ?
I haven't found where to change that. I didn't find the documentation
for doing the proper build,
src/README.building talks about doing the UW build.

I've successfully compiled boost 1.54.0 with

$ ./bootstrap.sh --with-python=/sw/python/3.4.4/bin/python3
--with-python-root=/sw/python/3.4.4 \

--libdir=/sw/python/3.4.4/lib--includedir=/sw/python/3.4.4/include/python3.4m

$ ./b2

My host is a CentOS 6.7 with python 3.4.4 compiled from source.

Thank you for your help,

Aki

On 2016-01-20 16:51, Brian Bockelman wrote:
Hi Aki,

The python bindings shipped by the HTCondor team indeed are compiled
against python2.

However, you can recompile HTCondor against your own version of boost
and python3 (I believe Ubuntu does this?): this is referred to a
âproperâ build versus a âUWâ build (where the dependencies are
downloaded by the build process).

Compilation with python3 is known to work, but I suspect thereâs
probably still undiscovered bugs because - to the best of myknowledge
- there are no heavy users.

Brian
On Jan 18, 2016, at 2:44 PM, Aki Ketolainen <akik@xxxxxxxxxxx>wrote:
Hi,
I'd like to start using htcondor python bindings for python3.Currently in v8.4.3 the bindings seem to be for python2
(lib/python/htcondor.so and lib/python/classad.so).
I found a mention of something related to this athttps://lists.cs.wisc.edu/archive/htcondor-devel/2014-January/msg00000.shtmlwhere it says:"Unfortunately, boost 1.49 does not support python3." I see boost1.49.0 being downloaded and built:
   src='http://parrot.cs.wisc.edu/externals/boost_1_49_0.tar.gz'
dst='/home/username/Downloads/htcondor/bld_external/boost_1_49_0-p3/dl/boost_1_49_0.tar.gz'
Could someone tell me what needs to be done for the python bindingsfor python3?
My host is a CentOS 6.7 with python 3.4.4 compiled from source.

Thank you,

Aki

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxxwith a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/


_______________________________________________
HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxxwith a

subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

References:
- [HTCondor-users] htcondor python bindings for python3 ?
  - From: Aki Ketolainen
- Re: [HTCondor-users] htcondor python bindings for python3 ?
  - From: Brian Bockelman
- Re: [HTCondor-users] htcondor python bindings for python3 ?
  - From: Aki Ketolainen

Prev by Date: Re: [HTCondor-users] UNC and mapped drive misunderstanding (not blocking issue)
Next by Date: [HTCondor-users] Question about ClassAd attributes on events log
Previous by thread: Re: [HTCondor-users] htcondor python bindings for python3 ?
Next by thread: [HTCondor-users] Does RemoteWallClockTime ever change while the job is running?
Index(es):
- Date
- Thread

Mailing List Archives

Public Access

Re: [HTCondor-users] htcondor python bindings for python3 ?