boost python to/from python type conversion

[James Amundson-2]

Hi,

I'm trying to write to/from type converters for my code, and I'm stuck.
My case doesn't map well to any of those in the boost python
documentation. I'm trying to pass objects of type MPI_Comm (really a C
type!) back and forth with mpi4py (which uses cython internally.) The
mpi4py author has provided a working example using manual wrapping. I am
attaching it to the end of this message. On the C++ side I can extract a
point to the MPI_Comm object from a Python object like so:

PyObject* py_obj;
// obviously, the value of py_obj has to be set somewhere...
MPI_Comm *comm_p = PyMPIComm_Get(py_obj);

I thought it would be simple to write a type converter to do that, but I
don't understand the mechanics. This example doesn't seem to map onto
the lvalue_from_pytype paradigm, or the scitbx container_conversion
paradigm.

Any suggestions?

Thanks,
Jim Amundson

-----------------------------------
// working example with manual conversion

#include <mpi.h>
#include <iostream>

static void sayhello(MPI_Comm comm)
{
   if (comm == MPI_COMM_NULL) {
     std::cout << "You passed MPI_COMM_NULL !!!" << std::endl;
     return;
   }
   int size;
   MPI_Comm_size(comm, &size);
   int rank;
   MPI_Comm_rank(comm, &rank);
   int plen; char pname[MPI_MAX_PROCESSOR_NAME];
   MPI_Get_processor_name(pname, &plen);
   std::cout <<
     "Hello, World! " <<
     "I am process " << rank <<
     " of " << size <<
     " on  " << pname <<
     "." << std::endl;
}


#include <boost/python.hpp>
#include <mpi4py/mpi4py.h>
using namespace boost::python;

static void hw_sayhello(object py_comm)
{
   PyObject* py_obj = py_comm.ptr();
   MPI_Comm *comm_p = PyMPIComm_Get(py_obj);
   if (comm_p == NULL) throw_error_already_set();
   sayhello(*comm_p);
}

BOOST_PYTHON_MODULE(helloworld)
{
   if (import_mpi4py() < 0) return; /* Python 2.X */

   def("sayhello", hw_sayhello);
}



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Ralf W. Grosse-Kunstleve]

lvalue_from_pytype only works for passing by value or const references/pointers, but that seems to match your case,
since you MPI_Comm is passed by value to your sayhello().

However, wrapping extern "C" isn't portable, AFAIK. You'll need a thin C++ wrapper for each of the C functions.

Did you consider writing a thin object-oriented wrapper around the raw MPI interfaces? Then it should be very straightforward to bind the wrapper with Boost.Python, and you'll have a more intuitive Python interface.
E.g. make a class that owns MPI_Comm, passed as boost::python::object, and then give it a .sayhello() method using the comm data member.




----- Original Message ----
From: James Amundson <amundson@...>
To: cplusplus-sig@...
Sent: Wednesday, September 30, 2009 2:40:29 PM
Subject: [C++-sig] boost python to/from python type conversion

Hi,

I'm trying to write to/from type converters for my code, and I'm stuck. My case doesn't map well to any of those in the boost python documentation. I'm trying to pass objects of type MPI_Comm (really a C type!) back and forth with mpi4py (which uses cython internally.) The mpi4py author has provided a working example using manual wrapping. I am attaching it to the end of this message. On the C++ side I can extract a point to the MPI_Comm object from a Python object like so:

PyObject* py_obj;
// obviously, the value of py_obj has to be set somewhere...
MPI_Comm *comm_p = PyMPIComm_Get(py_obj);

I thought it would be simple to write a type converter to do that, but I don't understand the mechanics. This example doesn't seem to map onto the lvalue_from_pytype paradigm, or the scitbx container_conversion paradigm.

Any suggestions?

Thanks,
Jim Amundson

-----------------------------------
// working example with manual conversion

#include <mpi.h>
#include <iostream>

static void sayhello(MPI_Comm comm)
{
  if (comm == MPI_COMM_NULL) {
    std::cout << "You passed MPI_COMM_NULL !!!" << std::endl;
    return;
  }
  int size;
  MPI_Comm_size(comm, &size);
  int rank;
  MPI_Comm_rank(comm, &rank);
  int plen; char pname[MPI_MAX_PROCESSOR_NAME];
  MPI_Get_processor_name(pname, &plen);
  std::cout <<
    "Hello, World! " <<
    "I am process " << rank <<
    " of " << size <<
    " on  " << pname <<
    "." << std::endl;
}


#include <boost/python.hpp>
#include <mpi4py/mpi4py.h>
using namespace boost::python;

static void hw_sayhello(object py_comm)
{
  PyObject* py_obj = py_comm.ptr();
  MPI_Comm *comm_p = PyMPIComm_Get(py_obj);
  if (comm_p == NULL) throw_error_already_set();
  sayhello(*comm_p);
}

BOOST_PYTHON_MODULE(helloworld)
{
  if (import_mpi4py() < 0) return; /* Python 2.X */

  def("sayhello", hw_sayhello);
}



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[Austin Bingham]

If it's any help, I wrote a small bit on the details of writing
converters. It might help clarify some of the mechanics which, I
agree, are a bit mysterious:

  http://misspent.wordpress.com/2009/09/27/how-to-write-boost-python-converters/

Austin

On Wed, Sep 30, 2009 at 11:40 PM, James Amundson <amundson@...> wrote:

> Hi,
>
> I'm trying to write to/from type converters for my code, and I'm stuck. My
> case doesn't map well to any of those in the boost python documentation. I'm
> trying to pass objects of type MPI_Comm (really a C type!) back and forth
> with mpi4py (which uses cython internally.) The mpi4py author has provided a
> working example using manual wrapping. I am attaching it to the end of this
> message. On the C++ side I can extract a point to the MPI_Comm object from a
> Python object like so:
>
> PyObject* py_obj;
> // obviously, the value of py_obj has to be set somewhere...
> MPI_Comm *comm_p = PyMPIComm_Get(py_obj);
>
> I thought it would be simple to write a type converter to do that, but I
> don't understand the mechanics. This example doesn't seem to map onto the
> lvalue_from_pytype paradigm, or the scitbx container_conversion paradigm.
>
> Any suggestions?
>
> Thanks,
> Jim Amundson
>
> -----------------------------------
> // working example with manual conversion
>
> #include <mpi.h>
> #include <iostream>
>
> static void sayhello(MPI_Comm comm)
> {
>  if (comm == MPI_COMM_NULL) {
>    std::cout << "You passed MPI_COMM_NULL !!!" << std::endl;
>    return;
>  }
>  int size;
>  MPI_Comm_size(comm, &size);
>  int rank;
>  MPI_Comm_rank(comm, &rank);
>  int plen; char pname[MPI_MAX_PROCESSOR_NAME];
>  MPI_Get_processor_name(pname, &plen);
>  std::cout <<
>    "Hello, World! " <<
>    "I am process " << rank <<
>    " of " << size <<
>    " on  " << pname <<
>    "." << std::endl;
> }
>
>
> #include <boost/python.hpp>
> #include <mpi4py/mpi4py.h>
> using namespace boost::python;
>
> static void hw_sayhello(object py_comm)
> {
>  PyObject* py_obj = py_comm.ptr();
>  MPI_Comm *comm_p = PyMPIComm_Get(py_obj);
>  if (comm_p == NULL) throw_error_already_set();
>  sayhello(*comm_p);
> }
>
> BOOST_PYTHON_MODULE(helloworld)
> {
>  if (import_mpi4py() < 0) return; /* Python 2.X */
>
>  def("sayhello", hw_sayhello);
> }
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig@...
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[James Amundson-2]

On 09/30/2009 11:48 PM, Ralf W. Grosse-Kunstleve wrote:
> lvalue_from_pytype only works for passing by value or const references/pointers, but that seems to match your case,
> since you MPI_Comm is passed by value to your sayhello().
>
> However, wrapping extern "C" isn't portable, AFAIK. You'll need a thin C++ wrapper for each of the C functions.
>    
Yes. I think you are right. I'm going to try that now.
> Did you consider writing a thin object-oriented wrapper around the raw MPI interfaces? Then it should be very straightforward to bind the wrapper with Boost.Python, and you'll have a more intuitive Python interface.
> E.g. make a class that owns MPI_Comm, passed as boost::python::object, and then give it a .sayhello() method using the comm data member.
>    

It's more complicated than that. mpi4py provides a nice object-oriented
wrapper on the Python side. The mpi4py wrapper is based on the
"standard" MPI C++ bindings. Unfortunately, older versions of MPI don't
support the C++ bindings and MPI has decided that the bindings will be
dropped in the future, so I really don't want to start using them now. I
think I will do as you suggest and just write very thin wrappers around
the C interfaces.

Thanks for your help.

--Jim Amundson

_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig

[James Amundson-2]

On 10/01/2009 01:22 AM, Austin Bingham wrote:
> If it's any help, I wrote a small bit on the details of writing
> converters. It might help clarify some of the mechanics which, I
> agree, are a bit mysterious:
>
>    http://misspent.wordpress.com/2009/09/27/how-to-write-boost-python-converters/
>
>    
Wonderful. Your blog post is very helpful. I wish you had written it
(and I had found it!) a long time ago.

Thanks,
Jim Amundson

_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@...
http://mail.python.org/mailman/listinfo/cplusplus-sig