Async DMA question regarding dma_async_memcpy_buf_to_buf

[Bruce_Leonard]

I have a question about
.../drivers/dma/dmaengine.c/dma_async_memcpy_buf_to_buf() in 2.6.28-rc2.
It looks like it always assumes that the source pointer points to data
coming from the CPU and the destination pointer always points to the DMA
device.  I say this because of the following two lines from the function:

 dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE);
 dma_dest = dma_map_single(dev->dev, dest, len, DMA_FROM_DEVICE);


As you can see 'src' is mapped DMA_TO_DEVICE indicating the data is going
from the CPU to the device, and 'dest' is mapped DMA_FROM_DEVICE,
indicating the opposite direction.  Seems to me this means the function is

assuming a certain direction (i.e., a write to the device), but there
isn't a corresponding function for going the other direction.  This leads
me to believe that the function is intended to operate in either
direction.  But this is in contradiction to both the mapped directions and

what I've read in LDD3, ch 15.

The hardware doesn't care (I'm using an MPC8347E), as far as the DMA
engine is concerned these are just addresses.  All of this goes to cache
coherency.  The map/unmap functions are supposed to ensure that data is
copied and caches flushed at the right times to ensure cache coherency. So

the question is this; is the dma_async_memcpy_buf_to_buf() function
intended to be bi-directional and is it safe to pass it a 'src' pointer
that's actually coming from the device, or does there need to be a second
function for doing a copy from the device to the CPU?

Thanks.

Bruce
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Scott Wood-2]

Bruce_Leonard@... wrote:
> The hardware doesn't care (I'm using an MPC8347E), as far as the DMA
> engine is concerned these are just addresses.  All of this goes to cache
> coherency.  The map/unmap functions are supposed to ensure that data is
> copied and caches flushed at the right times to ensure cache coherency. So
>
> the question is this; is the dma_async_memcpy_buf_to_buf() function
> intended to be bi-directional and is it safe to pass it a 'src' pointer
> that's actually coming from the device, or does there need to be a second
> function for doing a copy from the device to the CPU?

The "device" is the DMA engine, thus src is inherently going to the
device and dest is inherently coming from the device.

-Scott
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Bruce_Leonard]

Hi Scott,

Thanks for the reply.

> >
> > the question is this; is the dma_async_memcpy_buf_to_buf() function
> > intended to be bi-directional and is it safe to pass it a 'src'
pointer
> > that's actually coming from the device, or does there need to be a
second
> > function for doing a copy from the device to the CPU?
>
> The "device" is the DMA engine, thus src is inherently going to the
> device and dest is inherently coming from the device.
>

Yes, I understand the directions and what they apply to, that's obvious
from the macros themselves.  What I don't understand is if it's safe, from
a kernel/cache standpoint, to pass to the function a src pointer coming
_from_ the device and a dest pointer that's going _to_ the device?  In
other words, can I use this function to _write_ to a DMA slave?  If not,
there's a functionality missing from the async DMA engine.

Thanks.

Bruce
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Scott Wood-2]

Bruce_Leonard@... wrote:
> Yes, I understand the directions and what they apply to, that's obvious
> from the macros themselves.  What I don't understand is if it's safe, from
> a kernel/cache standpoint, to pass to the function a src pointer coming
> _from_ the device and a dest pointer that's going _to_ the device?  In
> other words, can I use this function to _write_ to a DMA slave?  If not,
> there's a functionality missing from the async DMA engine.

What exactly do you mean by "the device"?  if it's the DMA engine, then
that's a meaningless request, by the definition of "source" and
"destination".  If you mean some other device that happens to be
providing a buffer for you to DMA into or out of, that's irrelevant to
the DMA API; it's just memory that happens to live somewhere else (and
possibly not be cached).

-Scott
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Bruce_Leonard]

>
> What exactly do you mean by "the device"?  if it's the DMA engine, then
> that's a meaningless request, by the definition of "source" and
> "destination".  If you mean some other device that happens to be
> providing a buffer for you to DMA into or out of, that's irrelevant to
> the DMA API; it's just memory that happens to live somewhere else (and
> possibly not be cached).
>

Hummm, let me try again, I may be tripping up on my ignorance of the
kernel.  According to Linux Device Drivers 3, the DMA_TO/FROM_DEVICE
macros impact when the dma_map/unmap_single functions copy data and do
cache flushes to ensure cache coherency.  DMA_TO_DEVICE tells
dma_map_single() to ensure that all data is copied out to memory and cache
is flushed before doing the transfer and DMA_FROM_DEVICE tell
dma_unmap_single() to ensure all data is in main memory after the
transfer.  (My understanding is that this is really only important on
archs that use bounce buffers which I'm not).

So, no as far as "the device" (and I do mean the DMA engine) is concerned,
the "mapping" of the src and dest pointers make zero difference, the
hardware doesn't care since it's just acting on raw addresses.  However,
it does (or at least I think it does) matter to the kernel and making sure
that the cache doesn't get screwed up.  That all being said, I may be
worried about nothing.  I don't know enough about any of this to be sure.
I do know that in the past I've spent weeks trying to track down problems
that were caused by cache coherency, and LDD3 is clearly warning about
cache problems if these macros aren't used correctly.  But it sounds like
you're saying those macros are unimportant.  The async DMA stuff is new
since LDD3, so do those macros not apply?  Can
dma_async_memcpy_buf_to_buf() _safely_ be used to DMA either direction
(CPU <=> peripheral) _without_ cache coherency problems?

Thanks for the help.

Bruce
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Scott Wood-2]

Bruce_Leonard@... wrote:
> Hummm, let me try again, I may be tripping up on my ignorance of the
> kernel.  According to Linux Device Drivers 3, the DMA_TO/FROM_DEVICE
> macros impact when the dma_map/unmap_single functions copy data and do
> cache flushes to ensure cache coherency.  DMA_TO_DEVICE tells
> dma_map_single() to ensure that all data is copied out to memory and cache
> is flushed before doing the transfer and DMA_FROM_DEVICE tell
> dma_unmap_single() to ensure all data is in main memory after the
> transfer.

Right.

> (My understanding is that this is really only important on
> archs that use bounce buffers which I'm not).

It's also important on architectures where DMA is non-coherent (mpc83xx
DMA is coherent).

> So, no as far as "the device" (and I do mean the DMA engine) is concerned,
> the "mapping" of the src and dest pointers make zero difference, the
> hardware doesn't care since it's just acting on raw addresses.  However,
> it does (or at least I think it does) matter to the kernel and making sure
> that the cache doesn't get screwed up.

Right.

> But it sounds like
> you're saying those macros are unimportant.

No, they're very important on certain hardware.

> Can dma_async_memcpy_buf_to_buf() _safely_ be used to DMA either direction
> (CPU <=> peripheral) _without_ cache coherency problems?

Any given memcpy operation involves *both* directions.  Data is copied
out of host memory into the DMA engine's internal buffer (this is the
src mapping), and then it its copied back into host memory at a
different address (this is the dest mapping).

-Scott
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Bruce_Leonard]

>
> Any given memcpy operation involves *both* directions.  Data is copied
> out of host memory into the DMA engine's internal buffer (this is the
> src mapping), and then it its copied back into host memory at a
> different address (this is the dest mapping).
>

Whoops!  Now we may be getting to the heart of what I'm confused about ;).
 My bad, some more background may be in order for you to help me.

What I'm trying to do is set up a DMA transfer to/from a NAND flash
controller hanging on a PCI bus.  So, let's say I'm doing a read of the 2K
NAND page.  My "source" is a PCI address and is what I was assuming I
needed to pass to the memcpy function as "src".  But that assumption
conflicts with your statement that "Data is copied out of host memory into
the DMA engine's internal buffer (this is the src mapping)".  Is this
where my ignorance is jumping up and bitting me?

Maybe (with you clarification) this is starting to make sense.  It doesn't
matter if the "src" is host memory or PCI space.  What the "src" and it's
DMA_TO_DEVICE mapping is saying to the kernel is "this data needs to be
moved into the DMA engine", while the "dest" and it's DMA_FROM_DEIVCE
mapping is telling the kernel "this data needs to be moved from the DMA
engine".

Or am I even more confused :(  ?

Bruce
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Scott Wood-2]

Bruce_Leonard@... wrote:
> Maybe (with you clarification) this is starting to make sense.  It doesn't
> matter if the "src" is host memory or PCI space.  What the "src" and it's
> DMA_TO_DEVICE mapping is saying to the kernel is "this data needs to be
> moved into the DMA engine", while the "dest" and it's DMA_FROM_DEIVCE
> mapping is telling the kernel "this data needs to be moved from the DMA
> engine".

Right.

-Scott
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded

[Bruce_Leonard]

Scott Wood <scottwood@...> wrote on 11/24/2008 01:28:20 PM:

> Bruce_Leonard@... wrote:
> > Maybe (with you clarification) this is starting to make sense.  It
doesn't
> > matter if the "src" is host memory or PCI space.  What the "src" and
it's
> > DMA_TO_DEVICE mapping is saying to the kernel is "this data needs to
be
> > moved into the DMA engine", while the "dest" and it's DMA_FROM_DEIVCE
> > mapping is telling the kernel "this data needs to be moved from the
DMA
> > engine".
>
> Right.
>
> -Scott

Thank you sooooo much.  Now it makes sense.  I guess I was getting bogged
down in the hardware sense of "src" and "dest"

Thanks again.

Bruce
_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@...
https://ozlabs.org/mailman/listinfo/linuxppc-embedded