FFTW python bindings again

[Jochen Schröder]

Hi all,
about starting a new thread, stupid gmail does not show my posts to the
list. Anyways I noticed a big mistake in how I was allocating the
aligned memory thus it was actually not guaranteed to be 16byte aligned.
I guess it didn't show up because I was mainly testing using complex
numbers and malloc just took the next free block, which happened to be
aligned because I had just allocated a large chunk of aligned data.
Anyways I have created a new version where the issue is fixed. It can
again be found at http://pyfftw.berlios.de.

Cheers
Jochen

P.S.: I haven't received any comments on this, is this not of interest
to the scipy community?


_______________________________________________
SciPy-user mailing list
SciPy-user@...
http://projects.scipy.org/mailman/listinfo/scipy-user

[Robert Kern-2]

On Mon, Jan 26, 2009 at 17:45, Jochen <cycomanic@...> wrote:
> Hi all,
> about starting a new thread, stupid gmail does not show my posts to the
> list.

Just reply to your message in Sent Mail.

> Anyways I noticed a big mistake in how I was allocating the
> aligned memory thus it was actually not guaranteed to be 16byte aligned.
> I guess it didn't show up because I was mainly testing using complex
> numbers and malloc just took the next free block, which happened to be
> aligned because I had just allocated a large chunk of aligned data.
> Anyways I have created a new version where the issue is fixed. It can
> again be found at http://pyfftw.berlios.de.
>
> Cheers
> Jochen
>
> P.S.: I haven't received any comments on this, is this not of interest
> to the scipy community?

Many people simply don't comment. Please do keep us informed, though!

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
SciPy-user mailing list
SciPy-user@...
http://projects.scipy.org/mailman/listinfo/scipy-user

[Gary Pajer-2]

On Mon, Jan 26, 2009 at 6:45 PM, Jochen <cycomanic@...> wrote:

Hi all,

[...]

Cheers
Jochen

P.S.: I haven't received any comments on this, is this not of interest
to the scipy community?

I'm interested.  In fact I downloaded it.  I've only taken a very quick look, but perhaps you can answer a question:  is this OS agnostic, or is it Linux only?

-gary

_______________________________________________
SciPy-user mailing list
SciPy-user@...
http://projects.scipy.org/mailman/listinfo/scipy-user

[Jochen Schröder]

On Mon, 2009-01-26 at 21:08 -0500, Gary Pajer wrote:

> On Mon, Jan 26, 2009 at 6:45 PM, Jochen <cycomanic@...> wrote:
>         Hi all,
>
> [...]
>
>        
>        
>         Cheers
>         Jochen
>        
>         P.S.: I haven't received any comments on this, is this not of
>         interest
>         to the scipy community?
>
> I'm interested.  In fact I downloaded it.  I've only taken a very
> quick look, but perhaps you can answer a question:  is this OS
> agnostic, or is it Linux only?
>
> -gary

This should be OS agnostic, however I have not tested on any other
systems (don't have a windows/OSX machine to easily test on). The only
thing that could fail is loading the fftw shared library.
I do a:
lib = ctypes.cdll.LoadLibrary(util.find_library('fftw3'))
for this to be successful ctypes needs to find the fftw3 library. The
way I understand the ctypes documentation this should also work in
Windows or OSX and other unicies. I also assumed that fftw3 uses c-type
calling conventions on all platforms.
I'm actually checking if ctypes can find the fftw3 libraries in setup.py
if not the install will fail. I would actually be grateful if you could
check. If you don't want to install anything you can just do a python
setup.py build, the setup will raise an exception if ctypes cannot find
fftw3. Providing the possibility for specifying the path to fftw3 is
actually on my TODO.

Thanks for the interest
Cheers
Jochen
> _______________________________________________
> SciPy-user mailing list
> SciPy-user@...
> http://projects.scipy.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-user mailing list
SciPy-user@...
http://projects.scipy.org/mailman/listinfo/scipy-user

[cdavid]

Jochen wrote:
> Hi all,
> about starting a new thread, stupid gmail does not show my posts to the
> list. Anyways I noticed a big mistake in how I was allocating the
> aligned memory thus it was actually not guaranteed to be 16byte aligned.
> I guess it didn't show up because I was mainly testing using complex
> numbers and malloc just took the next free block, which happened to be
> aligned because I had just allocated a large chunk of aligned data.
>  

I believe fftw automatically detects whether your array is aligned or
not - problem appear when you create your plan with aligned pointers,
but use other pointers later. That's one reason why fftw backend was not
that fast in scipy BTW, because the simplest way to ensure this was to
copy data into aligned buffers.

At least on linux, allocating big buffers with malloc is almost
guaranteed not to be aligned, because of its use of mmap above a certain
threshold. We discovered this fact a while ago:

http://projects.scipy.org/pipermail/scipy-dev/2007-August/007591.html

Those are some of the reasons why we decided to drop fftw support: to
use it efficiently is not that easy, because we would first need
guarantees about aligned allocator (once you take into accout that numpy
also uses realloc, just using posix_memalign is not enough).

On the other hand, I think it would be very nice to have fftw wrappers
outside scipy. For some technical aspects, I answered to you in your
other post,

David
_______________________________________________
SciPy-user mailing list
SciPy-user@...
http://projects.scipy.org/mailman/listinfo/scipy-user

[Jochen Schröder]

On Tue, 2009-01-27 at 12:43 +0900, David Cournapeau wrote:

> Jochen wrote:
> > Hi all,
> > about starting a new thread, stupid gmail does not show my posts to the
> > list. Anyways I noticed a big mistake in how I was allocating the
> > aligned memory thus it was actually not guaranteed to be 16byte aligned.
> > I guess it didn't show up because I was mainly testing using complex
> > numbers and malloc just took the next free block, which happened to be
> > aligned because I had just allocated a large chunk of aligned data.
> >  
>
> I believe fftw automatically detects whether your array is aligned or
> not - problem appear when you create your plan with aligned pointers,
> but use other pointers later. That's one reason why fftw backend was not
> that fast in scipy BTW, because the simplest way to ensure this was to
> copy data into aligned buffers.

Yes I understand that. I was using a somewhat hackish way of creating
the memory aligned array, i.e. I was casting the pointer returned from
ctypes to a bytes array and then passed that to ndarray.__new__ as a
buffer. I didn't realise was that in the process I was allocating new
memory, which when I tested manually was still aligned because I had
just allocated aligned array (I was only using small arrays). I now use
PyBuffer_FromReadWriteMemory to create a buffer object to pass to
ndarray.__new__ in order to create the aligned memory.


> At least on linux, allocating big buffers with malloc is almost
> guaranteed not to be aligned, because of its use of mmap above a certain
> threshold. We discovered this fact a while ago:
>
> http://projects.scipy.org/pipermail/scipy-dev/2007-August/007591.html
>
I think I stumbled accross that thread when I was looking for fftw
bindings.

> Those are some of the reasons why we decided to drop fftw support: to
> use it efficiently is not that easy, because we would first need
> guarantees about aligned allocator (once you take into accout that numpy
> also uses realloc, just using posix_memalign is not enough).
>
> On the other hand, I think it would be very nice to have fftw wrappers
> outside scipy. For some technical aspects, I answered to you in your
> other post,
>
> David

Thanks for the comments
Cheers
Jochen

> _______________________________________________
> SciPy-user mailing list
> SciPy-user@...
> http://projects.scipy.org/mailman/listinfo/scipy-user

_______________________________________________
SciPy-user mailing list
SciPy-user@...
http://projects.scipy.org/mailman/listinfo/scipy-user